Oct 31 03:12:45.118190 ip-10-0-129-132 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 31 03:12:45.118205 ip-10-0-129-132 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 31 03:12:45.118212 ip-10-0-129-132 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 31 03:12:45.118450 ip-10-0-129-132 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 31 03:12:55.197503 ip-10-0-129-132 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 31 03:12:55.197518 ip-10-0-129-132 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 255bf544b053406cbefd6ee08e6ecca2 --
Oct 31 03:14:42.310976 ip-10-0-129-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:14:42.766700 ip-10-0-129-132 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:14:42.766700 ip-10-0-129-132 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:14:42.766700 ip-10-0-129-132 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:14:42.766700 ip-10-0-129-132 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:14:42.771097 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771005    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.771097 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771096    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.771097 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771102    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771105    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771109    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771112    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771115    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771117    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771120    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771123    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771126    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771129    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771132    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771134    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771137    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771140    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771142    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771145    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771147    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771152    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771154    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771157    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.771200 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771160    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771162    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771165    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771172    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771175    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771178    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771182    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771187    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771190    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771193    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771195    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771198    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771201    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771204    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771207    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771210    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771213    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771216    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771219    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.771683 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771222    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771224    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771227    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771230    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771232    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771235    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771238    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771240    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771243    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771247    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771250    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771252    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771255    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771257    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771260    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771264    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771267    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771270    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771272    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771277    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.772126 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771282    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771285    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771287    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771290    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771293    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771295    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771298    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771301    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771303    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771306    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771308    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771311    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771314    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771316    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771318    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771321    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771324    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771326    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771329    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.772591 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771333    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771336    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771339    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771342    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771344    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771347    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771349    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771352    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771355    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771358    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771361    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771363    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771366    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771369    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771371    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771374    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771377    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771379    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771382    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771385    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.773098 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771388    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771391    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771394    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771397    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771399    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.771402    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772220    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772228    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772231    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772234    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772237    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772240    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772243    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772247    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772250    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772253    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772255    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772258    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772260    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772263    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.773559 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772266    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772269    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772271    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772274    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772277    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772279    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772282    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772284    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772287    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772290    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772292    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772295    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772298    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772300    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772303    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772306    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772308    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772311    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772313    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.774061 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772316    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772318    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772321    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772323    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772326    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772330    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772333    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772336    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772339    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772341    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772344    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772346    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772349    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772352    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772354    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772357    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772359    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772362    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772365    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772368    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.774512 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772370    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772373    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772375    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772378    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772381    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772383    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772386    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772390    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772392    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772395    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772420    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772424    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772429    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772433    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772437    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772439    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772442    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772446    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772449    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.775000 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772451    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772454    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772457    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772460    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772463    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772466    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772470    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772473    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772476    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772479    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772482    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772484    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772487    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772490    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772492    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772495    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772497    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772500    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772503    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772505    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.775461 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772508    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772510    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772513    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772515    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772518    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772521    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772523    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772526    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772528    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772531    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772535    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772538    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772541    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.772543    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773091    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773098    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773101    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773104    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773108    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773111    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.775953 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773114    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773117    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773120    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773122    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773125    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773127    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773130    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773132    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773135    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773137    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773140    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773143    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773146    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773148    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773166    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773171    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773174    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773177    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773180    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773184    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.776427 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773187    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773190    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773193    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773196    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773199    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773201    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773205    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773207    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773210    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773213    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773216    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773219    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773221    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773224    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773227    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773230    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773233    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773237    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773241    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.776930 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773243    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773246    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773249    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773251    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773254    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773257    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773259    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773262    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773265    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773268    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773270    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773273    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773276    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773278    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773281    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773283    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773287    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773290    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773294    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.777375 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773297    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773301    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773303    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773307    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773309    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773312    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773315    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773317    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773320    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773323    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773325    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773328    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773331    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773334    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773336    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773339    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773341    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773344    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773346    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773349    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.777888 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773352    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773355    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773357    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773360    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773363    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773365    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773368    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773370    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773374    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773376    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773379    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773381    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773384    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773386    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773389    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773393    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773396    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773398    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773401    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773403    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.778362 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773407    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.773409    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775338    2573 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775354    2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775362    2573 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775367    2573 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775372    2573 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775375    2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775380    2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775385    2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775388    2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775391    2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775395    2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775398    2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775402    2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775405    2573 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775408    2573 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775411    2573 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775414    2573 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775418    2573 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775422    2573 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775425    2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775428    2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775432    2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:14:42.778840 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775435    2573 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775440    2573 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775443    2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775449    2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775453    2573 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775456    2573 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775459    2573 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775462    2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775466    2573 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775470    2573 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775474    2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775477    2573 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775480    2573 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775483    2573 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775487    2573 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775490    2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775495    2573 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775499    2573 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775502    2573 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775505    2573 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775508    2573 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775513    2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775517    2573 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775520    2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775524    2573 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:14:42.779399 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775527    2573 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775530    2573 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775533    2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775537    2573 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775540    2573 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775543    2573 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775546    2573 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775550    2573 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775554    2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775558    2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775562    2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775565    2573 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775569    2573 flags.go:64] FLAG: --help="false"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775572    2573 flags.go:64] FLAG: --hostname-override="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775576    2573 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775579    2573 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775582    2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775585    2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775588    2573 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775591    2573 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775594    2573 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775597    2573 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775600    2573 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:14:42.780026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775614    2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775619    2573 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775624    2573 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775628    2573 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775631    2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775634    2573 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775637    2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775641    2573 flags.go:64] FLAG: --lock-file=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775643    2573 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775646    2573 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775650    2573 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775655    2573 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775659    2573 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775662    2573 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775665    2573 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775668    2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775671    2573 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775674    2573 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775677    2573 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775683    2573 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775686    2573 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775696    2573 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775700    2573 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775703    2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775706    2573 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:14:42.780584 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775709    2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775712    2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775715    2573 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775718    2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775730    2573 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775733    2573 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775736    2573 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775739    2573 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775742    2573 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775745    2573 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775748    2573 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775751    2573 flags.go:64] FLAG: --port="10250"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775755    2573 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775758    2573 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07b853e1a036cbd75"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775761    2573 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775764    2573 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775767    2573 flags.go:64] FLAG: --register-node="true"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775770    2573 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775774    2573 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775777    2573 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775780    2573 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775783    2573 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775787    2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775790    2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775793    2573 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775796    2573 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:14:42.781247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775799    2573 flags.go:64] FLAG: --runonce="false"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775802    2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775805    2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775809    2573 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775812    2573 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775815    2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775818    2573 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775821    2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775825    2573 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775827    2573 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775830    2573 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775833    2573 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775836    2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775840    2573 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775843    2573 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775846    2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775856    2573 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775859    2573 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775862    2573 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775866    2573 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775869    2573 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775872    2573 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775875    2573 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775878    2573 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775881    2573 flags.go:64] FLAG: --v="2"
Oct 31 03:14:42.781893 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775886    2573 flags.go:64] FLAG: --version="false"
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775891    2573 flags.go:64] FLAG: --vmodule=""
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775895    2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.775898    2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776045    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776049    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776052    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776055    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776058    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776061    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776064    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776067    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776070    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776073    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776075    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776078    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776081    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776083    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776086    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776088    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.782577 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776091    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776094    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776096    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776100    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776103    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776106    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776109    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776111    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776114    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776117    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776119    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776122    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776124    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776127    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776130    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776132    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776135    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776137    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776140    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776144    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.783081 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776147    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776149    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776152    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776155    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776157    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776160    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776163    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776166    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776169    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776172    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776175    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776177    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776180    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776183    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776185    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776189    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776192    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776195    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776197    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.783554 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776200    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776203    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776205    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776208    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776211    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776214    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776216    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776219    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776221    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776224    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776227    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776231    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776235    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776239    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776242    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776244    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776247    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776250    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776253    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.784028 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776255    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776258    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776261    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776264    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776268    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776272    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776275    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776278    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776281    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776284    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776287    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776290    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776292    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776295    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776298    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776300    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776303    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776305    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776308    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.784578 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776311    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776313    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776316    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776318    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776321    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776324    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776327    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776330    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776333    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776335    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776338    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776341    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776344    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.776371    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776408    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776412    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.785078 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776415    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776418    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776421    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776424    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776426    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776429    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776433    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776436    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776439    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776441    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776444    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776447    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776449    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776452    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776454    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776457    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776460    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776462    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776465    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776468    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.785457 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776470    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776473    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776475    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776480    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776483    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776485    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776488    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776490    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776493    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776495    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776498    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776501    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776504    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776506    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776509    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776512    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776514    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776517    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776521    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776523    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.785948 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776526    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776529    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776532    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776534    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776537    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776539    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776542    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776544    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776547    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776549    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776552    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776554    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776557    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776560    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776563    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776568    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776571    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776573    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776576    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.786417 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776579    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776581    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776584    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776586    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776590    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776598    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776601    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776617    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776620    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776623    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776626    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776630    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776632    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776635    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776638    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776640    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776643    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776647    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776651    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.786944 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776655    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776658    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776661    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776664    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776666    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776669    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776672    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776675    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776677    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776681    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776684    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776686    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776690    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776692    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776695    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776698    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776700    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776703    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776706    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776709    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.787385 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776712    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776714    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776717    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776720    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776723    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.776727    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.784303    2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.784320    2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.784338    2573 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.784343    2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784402    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784408    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784411    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784414    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784417    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784420    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784423    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784425    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784428    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.787891 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784431    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784434    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784438    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784444    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784447    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784450    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784453    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784456    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784459    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784462    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784465    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784469    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784473    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784477    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784480    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784483    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784487    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784490    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.788345 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784493    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784496    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784500    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784503    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784506    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784509    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784511    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784514    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784517    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784521    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784523    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784526    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784529    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784532    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784534    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784537    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784540    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784542    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784545    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784548    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.788792 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784550    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784553    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784556    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784558    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784562    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784565    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784568    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784571    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784574    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784577    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784580    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784584    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784586    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784589    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784592    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784595    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784598    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784600    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784617    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784620    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.789270 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784623    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784626    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784629    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784631    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784634    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784637    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784640    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784642    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784645    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784648    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784651    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784654    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784657    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784659    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784662    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784665    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784668    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784671    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784674    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784677    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.789765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784680    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784682    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784685    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784688    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784690    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784693    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784696    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784699    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784702    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784704    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784707    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784709    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784712    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784715    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784718    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784721    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784724    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784726    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.790263 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784729    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.784758    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784914    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784919    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784922    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784925    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784928    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784931    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784933    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784936    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784939    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784942    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784945    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784948    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784952    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784954    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.790821 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784957    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784960    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784962    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784965    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784967    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784970    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784972    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784975    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784978    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784980    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784983    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784986    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784988    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784991    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784993    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784996    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.784999    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785001    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785004    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785007    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.791209 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785010    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785012    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785015    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785017    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785020    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785023    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785025    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785028    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785032    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785035    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785039    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785044    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785047    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785049    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785052    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785054    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785057    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785059    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785062    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.791684 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785065    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785067    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785070    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785073    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785075    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785078    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785081    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785084    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785086    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785089    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785092    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785095    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785098    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785101    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785106    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785109    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785112    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785114    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785117    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.792142 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785120    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785122    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785125    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785128    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785131    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785134    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785137    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785140    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785143    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785146    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785148    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785151    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785154    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785156    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785159    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785162    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785164    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785167    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785170    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.792623 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785172    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785175    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785178    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785180    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785183    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785186    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785189    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785191    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785194    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785196    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785199    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785202    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785204    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785208    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785210    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.785233    2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:14:42.793085 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785267    2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785271    2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785274    2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785277    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785280    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785283    2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785286    2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785289    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785292    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785295    2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785297    2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785300    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785303    2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785305    2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785308    2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785311    2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785313    2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785316    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785319    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:14:42.793757 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785321    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785324    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785326    2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785330    2573 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785332    2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785335    2573 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785338    2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785340    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785343    2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785345    2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785348    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785351    2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785354    2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785356    2573 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785359    2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785362    2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785364    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785367    2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785370    2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785373    2573 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:14:42.794462 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785376    2573 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785378    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785381    2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785383    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785386    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785388    2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785391    2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785393    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785396    2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785398    2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785401    2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785403    2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785406    2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785409    2573 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785412    2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785414    2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785417    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785419    2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785422    2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785424    2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:14:42.795103 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785427    2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785431    2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785435    2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785441    2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785444    2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785446    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785449    2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785451    2573 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785454    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785457    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785459    2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785463    2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785466    2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785468    2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785471    2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785474    2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785478    2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785480    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785483    2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:14:42.795788 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785485    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785488    2573 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785491    2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785493    2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785497    2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785500    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785503    2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785506    2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785509    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785512    2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785515    2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785517    2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785520    2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785522    2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785525    2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785527    2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785531    2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785534    2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785536    2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:14:42.796258 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785539    2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785542    2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785544    2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785547    2573 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785550    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785553    2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785557    2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785559    2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:42.785562    2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.785636    2573 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.789875    2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.790969    2573 server.go:1014] "Starting client certificate rotation"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.791073    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.791125    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.794008    2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:14:42.796717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.796643    2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:14:42.807812 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.807789    2573 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:14:42.824545 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.824514    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:14:42.866047 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.866013    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7md48"
Oct 31 03:14:42.871741 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.871718    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7md48"
Oct 31 03:14:42.929898 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.929637    2573 log.go:25] "Validated CRI v1 image API"
Oct 31 03:14:42.931292 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.931260    2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:14:42.935137 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.935110    2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ee241086-c6f9-42c0-8605-e6060086377c:/dev/nvme0n1p4 f8785297-4947-4f7a-bfd3-c17955ae6693:/dev/nvme0n1p3]
Oct 31 03:14:42.935229 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.935132    2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 03:14:42.941150 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.941012    2573 manager.go:217] Machine: {Timestamp:2026-10-31 03:14:42.939567482 +0000 UTC m=+0.498049349 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093515 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2731b5aae8aeb67acf3631b2fb3764 SystemUUID:ec2731b5-aae8-aeb6-7acf-3631b2fb3764 BootID:255bf544-b053-406c-befd-6ee08e6ecca2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:df:a9:bc:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:df:a9:bc:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:fe:4f:bc:04:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:14:42.941150 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.941135    2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:14:42.941316 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.941222    2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:14:42.944557 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944521    2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:14:42.944732 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944559    2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-132.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:14:42.944800 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944739    2573 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:14:42.944800 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944747    2573 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:14:42.944800 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944787    2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:14:42.944909 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.944806    2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:14:42.946035 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.946021    2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:14:42.950461 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950444    2573 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:14:42.950592 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950581    2573 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:14:42.950650 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950634    2573 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:14:42.950650 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950649    2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:14:42.950724 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950661    2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:14:42.950724 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950670    2573 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:14:42.950724 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.950679    2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:14:42.951052 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.951038    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:14:42.956016 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.955984    2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:14:42.962791 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.962777    2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:14:42.963822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.963809    2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:14:42.963863 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.963840    2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:14:42.964020 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964010    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:14:42.964049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964027    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:14:42.964049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964034    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:14:42.964049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964040    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:14:42.964049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964045    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:14:42.964151 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964052    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:14:42.964747 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964739    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:14:42.964781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964750    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:14:42.964781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964757    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:14:42.964781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964764    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:14:42.964781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964773    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:14:42.964781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.964781    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:14:42.965502 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.965491    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:14:42.965534 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.965504    2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:14:42.968660 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.968645    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:42.968945 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.968931    2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:42.969838 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.969826    2573 server.go:1263] "Started kubelet"
Oct 31 03:14:42.969942 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.969908    2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:14:42.970017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.969937    2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:14:42.970064 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.970045    2573 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:14:42.970755 ip-10-0-129-132 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:14:42.971517 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.971455    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-132.us-east-2.compute.internal" not found
Oct 31 03:14:42.972138 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.972117    2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:14:42.972745 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.972726    2573 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:14:42.976390 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:42.976371    2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 03:14:42.977829 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.977811    2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:14:42.979128 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979114    2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:14:42.979231 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:42.979212    2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-132.us-east-2.compute.internal\" not found"
Oct 31 03:14:42.979843 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979828    2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:14:42.979931 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979851    2573 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:14:42.979931 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979862    2573 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:14:42.980022 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979969    2573 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:14:42.980022 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979978    2573 factory.go:55] Registering systemd factory
Oct 31 03:14:42.980022 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979994    2573 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:14:42.980145 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.979980    2573 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:14:42.980219 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.980206    2573 factory.go:153] Registering CRI-O factory
Oct 31 03:14:42.980267 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.980220    2573 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:14:42.980309 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.980267    2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:14:42.980309 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.980281    2573 factory.go:103] Registering Raw factory
Oct 31 03:14:42.980309 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.980292    2573 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:14:42.981257 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.981238    2573 manager.go:319] Starting recovery of all containers
Oct 31 03:14:42.982532 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.982507    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:42.985245 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:42.985145    2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-132.us-east-2.compute.internal\" not found" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:42.985305 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.985247    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-132.us-east-2.compute.internal" not found
Oct 31 03:14:42.994855 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.994840    2573 manager.go:324] Recovery completed
Oct 31 03:14:42.996258 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:42.996230    2573 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 03:14:42.999252 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:42.999239    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.001124 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001106    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.001210 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001138    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.001210 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001148    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.001743 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001726    2573 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:14:43.001743 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001740    2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:14:43.001903 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001759    2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:14:43.001903 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.001733    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.002490 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.002470    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.002573 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.002504    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.002573 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.002517    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.004017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.004000    2573 policy_none.go:50] "Start"
Oct 31 03:14:43.004017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.004019    2573 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:14:43.004105 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.004030    2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:14:43.006045 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.006034    2573 policy_none.go:44] "Start"
Oct 31 03:14:43.057529 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.041423    2573 nodeinfomanager.go:422] Failed to publish CSINode: nodes "ip-10-0-129-132.us-east-2.compute.internal" not found
Oct 31 03:14:43.058397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.058359    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:14:43.058512 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.058358    2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:14:43.059980 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.059958    2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:14:43.060103 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.059991    2573 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:14:43.060103 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.060015    2573 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:14:43.060103 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.060052    2573 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:14:43.063727 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.063711    2573 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:14:43.063837 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.063745    2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:14:43.063837 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.063755    2573 server.go:87] "Starting device plugin registration server"
Oct 31 03:14:43.064033 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064020    2573 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:14:43.064033 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064028    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:43.064119 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064034    2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:14:43.064172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064131    2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:14:43.064242 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064207    2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:14:43.064242 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.064215    2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:14:43.064718 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.064694    2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:14:43.064803 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.064739    2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-132.us-east-2.compute.internal\" not found"
Oct 31 03:14:43.160758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.160708    2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"]
Oct 31 03:14:43.160917 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.160800    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.161832 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.161814    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.161935 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.161851    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.161935 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.161865    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.164170 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164151    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.164170 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164169    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.164347 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164333    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.164404 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164368    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.164989 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164970    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.165066 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164998    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.165066 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165011    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.165066 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165018    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.165066 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165041    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.165066 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165052    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.165209 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165071    2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.165209 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.164977    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.165209 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165181    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.165209 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.165194    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.167076 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.167062    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.167137 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.167088    2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:14:43.167841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.167828    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:14:43.167912 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.167852    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:14:43.167912 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.167867    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:14:43.173773 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.173757    2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.179572 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.179554    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.199542 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.199524    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.202427 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.202407    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.281045 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.281010    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.281196 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.281090    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.281196 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.281125    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.284482 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.284440    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:14:43.288264 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.288246    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:14:43.294747 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.294732    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:14:43.294827 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.294771    2573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.294827 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.294783    2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.312019 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.311992    2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:14:43.312140 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.312048    2573 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382345 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382319    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382415 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382357    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382415 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382382    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382480 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382445    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382480 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382455    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.382546 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.382446    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.590713 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.590602    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.593994 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.593972    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:14:43.790671 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.790630    2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 03:14:43.791198 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.790815    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:14:43.791198 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.790815    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:14:43.791198 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.790823    2573 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 31 03:14:43.879309 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.879195    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:09:43 +0000 UTC" deadline="2028-05-07 06:08:25.206024567 +0000 UTC"
Oct 31 03:14:43.879309 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.879252    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13298h53m41.326775166s"
Oct 31 03:14:43.952692 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.952652    2573 apiserver.go:52] "Watching apiserver"
Oct 31 03:14:43.960943 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.960921    2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:14:43.961244 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.961227    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-bjxm9","kube-system/konnectivity-agent-6csjw","openshift-cluster-node-tuning-operator/tuned-jmgbj","openshift-dns/node-resolver-nnxgp","openshift-multus/multus-7p689","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7","openshift-multus/network-metrics-daemon-bmxnp","openshift-network-diagnostics/network-check-target-4ww6v","openshift-image-registry/node-ca-wtqp4","openshift-ovn-kubernetes/ovnkube-node-5fgwn","kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-jjjbw","openshift-network-operator/iptables-alerter-xrkgp"]
Oct 31 03:14:43.963847 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.963824    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:43.963968 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.963903    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:43.965992 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.965979    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:43.968311 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.968283    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.969397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.969371    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:14:43.969496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.969427    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\""
Oct 31 03:14:43.969496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.969431    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:14:43.970537 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.970518    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:43.972923 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.971285    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:14:43.972923 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.971385    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\""
Oct 31 03:14:43.972923 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.971556    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:14:43.974034 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.974016    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:14:43.974092 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.974057    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\""
Oct 31 03:14:43.974133 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.974087    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:14:43.974199 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.974186    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.976443 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.976425    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.977429 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.977409    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:14:43.977429 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.977418    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:14:43.977568 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.977422    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:14:43.977568 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.977470    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:14:43.977698 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.977579    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\""
Oct 31 03:14:43.978599 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.978588    2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 03:14:43.978757 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.978741    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:43.978839 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.978815    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:43.979428 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.979413    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:14:43.979428 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.979425    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:14:43.979657 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.979644    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\""
Oct 31 03:14:43.979902 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.979888    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:14:43.980928 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.980910    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:43.981014 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:43.980962    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:43.983254 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.983235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:43.985834 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.985811    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:43.986100 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986117    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986145    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvnt\" (UniqueName: \"kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986162    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986180    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986196    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:43.986213 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986211    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dhc\" (UniqueName: \"kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986228    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986273    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986311    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986348    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986364    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986411    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxzfg\" (UniqueName: \"kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986455    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986474    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986491    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986521 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986516    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986531    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986546    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986560    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986583    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986601    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986671    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986720    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986757    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986790    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986827    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986851    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:43.986876 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986868    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986896    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.986944    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987000    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987027    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987133    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5lzv\" (UniqueName: \"kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987155    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987174    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987198    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987213    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987233    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987250    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987394 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987329    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987415    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987473    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987502    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987530    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcz65\" (UniqueName: \"kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987565    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:43.987884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.987604    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:43.988289 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.988273    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:43.990833 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:43.990814    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.019845 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.019823    2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 03:14:44.019963 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.019881    2573 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n5558"
Oct 31 03:14:44.027750 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.027730    2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n5558"
Oct 31 03:14:44.078061 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078022    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:14:44.078212 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078111    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:14:44.078212 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078201    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:14:44.078330 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078239    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:14:44.078330 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078252    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:14:44.078330 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078287    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:14:44.078530 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.078507    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:14:44.079017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079001    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:14:44.079054 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079031    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:14:44.079100 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079065    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:14:44.079100 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079059    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:14:44.079200 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079142    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\""
Oct 31 03:14:44.079329 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079311    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:14:44.079329 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079323    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:14:44.079425 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079350    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\""
Oct 31 03:14:44.079425 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.079375    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\""
Oct 31 03:14:44.080390 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.080364    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\""
Oct 31 03:14:44.080497 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.080484    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:14:44.080958 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.080941    2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:14:44.088114 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088094    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:44.088225 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088128    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.088225 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088155    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.088225 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088200    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088228    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088250    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.088269    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088298    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088314    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088329    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.088369    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:44.588325774 +0000 UTC m=+2.146807645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088368    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.088392 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088390    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088451    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlfgs\" (UniqueName: \"kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088483    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kth74\" (UniqueName: \"kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088537    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088597    2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088632    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088669    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088706    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.088821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088776    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088830    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088898    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088914    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088990    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.088999    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089012    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089021    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089077    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089134    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089168    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089192    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089203    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089221    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.089279 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089233    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089236    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089284    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089302    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089267    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089316    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089341    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089369    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089392    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089446    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5ktq\" (UniqueName: \"kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089505    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089508    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089508    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089551    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089580    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089609    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:44.090115 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089702    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089704    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089756    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089820    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089869    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089916    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089935    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.089982    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090024    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090052    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090078    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090095    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090177    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090196    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090227    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090231    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090255    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.091017 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090296    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090319    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090339    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090350    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090362    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090351    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090389    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090446    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090448    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-562d5\" (UniqueName: \"kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090519    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090532    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090620    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090691    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090707    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090732    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090749    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.091965 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lzv\" (UniqueName: \"kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090786    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090837    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090863    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090884    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090902    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090929    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.090973    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.091017    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091031    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091051    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.091091    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:44.591074413 +0000 UTC m=+2.149556287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091148    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091173    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091223    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091283    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091316    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz65\" (UniqueName: \"kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.092788 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091344    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091398    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091458    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091495    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091568    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091599    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091669    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvnt\" (UniqueName: \"kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091704    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091747    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dhc\" (UniqueName: \"kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091781    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzfg\" (UniqueName: \"kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091833    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.091948    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.092044    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.092665    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.092686    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.093359 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.093089    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:44.096286 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.096271    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:44.096369 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.096290    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:44.096369 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.096300    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:44.096369 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.096355    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:44.596342306 +0000 UTC m=+2.154824182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:44.100201 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.100174    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lzv\" (UniqueName: \"kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.101222 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.101205    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz65\" (UniqueName: \"kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.105523 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.105500    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzfg\" (UniqueName: \"kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:44.105871 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.105854    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvnt\" (UniqueName: \"kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.106123 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.106104    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dhc\" (UniqueName: \"kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.193049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.192940    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.192981    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193002    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193021    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193039    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193065    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193061    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193077    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193104    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193116    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193149    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193176    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193197    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193216    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193239    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193258    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ktq\" (UniqueName: \"kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193284    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193306    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193360    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193396    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193430    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.193436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193437    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193465    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193496    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193524    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562d5\" (UniqueName: \"kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193556    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193612    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193653    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193658    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193695    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193691    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193731    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193750    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193770    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193778    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193771    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193822    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193868    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194172 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193881    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193900    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193903    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193917    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193933    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193964    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193974    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.193992    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194037    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194057    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfgs\" (UniqueName: \"kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194076    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kth74\" (UniqueName: \"kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194233    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194305    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194441    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.194969 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.194675    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.195791 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.195769    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.195854 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.195835    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.195909 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.195854    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.195909 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.195862    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.196059 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.196032    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.196318 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.196300    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.196596 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.196580    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.202742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.202717    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ktq\" (UniqueName: \"kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.202939 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.202918    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562d5\" (UniqueName: \"kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.203094 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.203078    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kth74\" (UniqueName: \"kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.203878 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.203863    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfgs\" (UniqueName: \"kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.293649 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.293592    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:44.308362 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.308317    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:14:44.326027 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.325995    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:14:44.337065 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.337041    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7p689"
Oct 31 03:14:44.354205 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.354175    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:14:44.366171 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.366145    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:14:44.397585 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.397549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:14:44.411797 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.411755    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776064ad5d3782b191602f1d077b5a65.slice/crio-27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba WatchSource:0}: Error finding container 27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba: Status 404 returned error can't find the container with id 27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba
Oct 31 03:14:44.412732 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.412699    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79a249fe_49bc_4f12_b1f5_c2e4b3ecc990.slice/crio-448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271 WatchSource:0}: Error finding container 448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271: Status 404 returned error can't find the container with id 448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271
Oct 31 03:14:44.414903 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.414876    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbfbb892bae597c340f8a0c3dc3499e.slice/crio-f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61 WatchSource:0}: Error finding container f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61: Status 404 returned error can't find the container with id f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61
Oct 31 03:14:44.415575 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.415546    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:14:44.415752 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.415731    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod478f1160_d546_4149_aee6_7a3f6f3f520f.slice/crio-59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2 WatchSource:0}: Error finding container 59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2: Status 404 returned error can't find the container with id 59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2
Oct 31 03:14:44.418973 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.418956    2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:14:44.419040 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.418954    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:14:44.429642 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.429606    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85a63323_7c0f_462b_bdf1_f3a532f2d04c.slice/crio-2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475 WatchSource:0}: Error finding container 2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475: Status 404 returned error can't find the container with id 2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475
Oct 31 03:14:44.430646 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.430547    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3f060e6_cd83_4284_98ac_d39a6d8a01be.slice/crio-a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027 WatchSource:0}: Error finding container a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027: Status 404 returned error can't find the container with id a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027
Oct 31 03:14:44.597472 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.597386    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:44.597472 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.597436    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:44.597472 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.597460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:44.597705 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597559    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:44.597705 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597617    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:45.597601887 +0000 UTC m=+3.156083756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:44.597705 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597645    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:44.597705 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597672    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:44.597705 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597685    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:44.597870 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597727    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:45.597712325 +0000 UTC m=+3.156194184 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:44.597870 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597559    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:44.597870 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:44.597786    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:45.597774545 +0000 UTC m=+3.156256400 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:44.699080 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.699039    2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:44.763064 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.763036    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d23fdfb_3765_488e_bad2_d80a7fe3a162.slice/crio-dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19 WatchSource:0}: Error finding container dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19: Status 404 returned error can't find the container with id dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19
Oct 31 03:14:44.807151 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.807123    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3b2dbb0_3454_4290_81ef_3cf35b21c1d0.slice/crio-422cb1fdd69a529858971c6ff8bdcfa75a24566a0ab892ed034c3babb0a2742c WatchSource:0}: Error finding container 422cb1fdd69a529858971c6ff8bdcfa75a24566a0ab892ed034c3babb0a2742c: Status 404 returned error can't find the container with id 422cb1fdd69a529858971c6ff8bdcfa75a24566a0ab892ed034c3babb0a2742c
Oct 31 03:14:44.807907 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.807881    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1d993de_0f1b_4b88_a998_344eec8c23ac.slice/crio-381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a WatchSource:0}: Error finding container 381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a: Status 404 returned error can't find the container with id 381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a
Oct 31 03:14:44.809896 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:44.809868    2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:44.922880 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:44.922845    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod092c4d2f_ae5c_424d_b458_d19bf1e0e8f6.slice/crio-b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749 WatchSource:0}: Error finding container b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749: Status 404 returned error can't find the container with id b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749
Oct 31 03:14:45.027817 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:14:45.027782    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13947923_5437_4f19_b97d_c6e913cf8fc7.slice/crio-68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084 WatchSource:0}: Error finding container 68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084: Status 404 returned error can't find the container with id 68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084
Oct 31 03:14:45.030412 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.030355    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:44 +0000 UTC" deadline="2028-06-15 00:40:49.828873434 +0000 UTC"
Oct 31 03:14:45.030485 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.030413    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14229h26m4.798462982s"
Oct 31 03:14:45.049542 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.049508    2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:14:45.067019 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.066941    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7" event={"ID":"13947923-5437-4f19-b97d-c6e913cf8fc7","Type":"ContainerStarted","Data":"68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084"}
Oct 31 03:14:45.069177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.069087    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6csjw" event={"ID":"6d23fdfb-3765-488e-bad2-d80a7fe3a162","Type":"ContainerStarted","Data":"dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19"}
Oct 31 03:14:45.070890 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.070861    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerStarted","Data":"2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475"}
Oct 31 03:14:45.073300 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.073272    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerStarted","Data":"f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61"}
Oct 31 03:14:45.077822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.077792    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7p689" event={"ID":"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6","Type":"ContainerStarted","Data":"b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749"}
Oct 31 03:14:45.080578 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.080549    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnxgp" event={"ID":"b1d993de-0f1b-4b88-a998-344eec8c23ac","Type":"ContainerStarted","Data":"381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a"}
Oct 31 03:14:45.085040 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.085009    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jmgbj" event={"ID":"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0","Type":"ContainerStarted","Data":"422cb1fdd69a529858971c6ff8bdcfa75a24566a0ab892ed034c3babb0a2742c"}
Oct 31 03:14:45.089120 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.089095    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrkgp" event={"ID":"c3f060e6-cd83-4284-98ac-d39a6d8a01be","Type":"ContainerStarted","Data":"a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027"}
Oct 31 03:14:45.091781 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.091752    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2"}
Oct 31 03:14:45.094290 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.094263    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wtqp4" event={"ID":"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990","Type":"ContainerStarted","Data":"448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271"}
Oct 31 03:14:45.096225 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.096196    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"776064ad5d3782b191602f1d077b5a65","Type":"ContainerStarted","Data":"27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba"}
Oct 31 03:14:45.607155 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.607114    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:45.607397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.607197    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:45.607397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:45.607251    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:45.607533 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.607402    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:45.607533 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.607467    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:47.607449043 +0000 UTC m=+5.165930917 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:45.607899 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.607837    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:45.607999 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.607941    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:47.607923856 +0000 UTC m=+5.166405715 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:45.608062 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.608040    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:45.608062 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.608060    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:45.608173 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.608074    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:45.608173 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:45.608114    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:47.608100467 +0000 UTC m=+5.166582328 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:46.031430 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:46.031308    2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:44 +0000 UTC" deadline="2028-05-30 21:08:26.257170708 +0000 UTC"
Oct 31 03:14:46.031430 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:46.031381    2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13865h53m40.225795127s"
Oct 31 03:14:46.060798 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:46.060761    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:46.060992 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:46.060900    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:46.061662 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:46.061464    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:46.061662 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:46.061584    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:46.061890 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:46.061708    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:46.061890 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:46.061788    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:47.626578 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:47.626536    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:47.627277 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.626769    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:47.627277 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.626828    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:47.627277 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.626839    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:51.62682138 +0000 UTC m=+9.185303247 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:47.627277 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.626907    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:51.626887541 +0000 UTC m=+9.185369407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:47.627277 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:47.627233    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:47.627550 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:47.627321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:47.627550 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.627453    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:47.627550 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.627474    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:47.627550 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.627488    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:47.627550 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:47.627533    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:51.627519653 +0000 UTC m=+9.186001513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:48.060990    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:48.061096    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:48.061392    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:48.061471    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:48.061516    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:48.061685 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:48.061579    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:49.161499 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:49.160312    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="1be4dc60c0c2987477f4a9a94083e3df3c1fa541e12ccc6ae9a8240e24b92ac0" exitCode=0
Oct 31 03:14:49.161499 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:49.160596    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"1be4dc60c0c2987477f4a9a94083e3df3c1fa541e12ccc6ae9a8240e24b92ac0"}
Oct 31 03:14:50.060367 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.060321    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:50.060531 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:50.060465    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:50.060568 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.060549    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:50.060658 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:50.060638    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:50.060730 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.060698    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:50.060796 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:50.060778    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:50.185684 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.185647    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"776064ad5d3782b191602f1d077b5a65","Type":"ContainerStarted","Data":"1c8744fa95dc6ce578ecd40b6e873d41992aa8ea660339c4cd53dd5b972d3b15"}
Oct 31 03:14:50.188805 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.188766    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6csjw" event={"ID":"6d23fdfb-3765-488e-bad2-d80a7fe3a162","Type":"ContainerStarted","Data":"6b3de6249f92c24f32881b0678c1c3e029c67a52654d7d32f2475f1e2debb475"}
Oct 31 03:14:50.199466 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.199201    2573 generic.go:356] "Generic (PLEG): container finished" podID="dcbfbb892bae597c340f8a0c3dc3499e" containerID="2b35d65b1a4cdbae5039a6bc5a405516101081d294e8fa3140c4d48e448eede0" exitCode=0
Oct 31 03:14:50.199466 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.199253    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerDied","Data":"2b35d65b1a4cdbae5039a6bc5a405516101081d294e8fa3140c4d48e448eede0"}
Oct 31 03:14:50.216247 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.216170    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal" podStartSLOduration=7.216150202 podStartE2EDuration="7.216150202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:50.201359399 +0000 UTC m=+7.759841275" watchObservedRunningTime="2026-10-31 03:14:50.216150202 +0000 UTC m=+7.774632081"
Oct 31 03:14:50.228886 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:50.228837    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6csjw" podStartSLOduration=2.533568027 podStartE2EDuration="7.228818216s" totalImagesPullingTime="4.695250189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:50.228732129 +0000 UTC m=+7.787214005" watchObservedRunningTime="2026-10-31 03:14:50.228818216 +0000 UTC m=+7.787300070"
Oct 31 03:14:51.204475 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.204432    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wtqp4" event={"ID":"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990","Type":"ContainerStarted","Data":"e4fb2c1ae11c54b0a70f22391892dd63f47f5fd538d8dc6dff382e6b153a4ad9"}
Oct 31 03:14:51.208366 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.208329    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerStarted","Data":"b49d20fc9ddacadc24b7e0c2aeb106b1ee0bc9da529d873b0112d46326c1182b"}
Oct 31 03:14:51.244713 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.244647    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wtqp4" podStartSLOduration=2.397968621 podStartE2EDuration="8.244626423s" totalImagesPullingTime="5.846657802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:51.225316148 +0000 UTC m=+8.783798024" watchObservedRunningTime="2026-10-31 03:14:51.244626423 +0000 UTC m=+8.803108296"
Oct 31 03:14:51.245026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.244989    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" podStartSLOduration=8.244981065 podStartE2EDuration="8.244981065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:51.244164732 +0000 UTC m=+8.802646602" watchObservedRunningTime="2026-10-31 03:14:51.244981065 +0000 UTC m=+8.803462944"
Oct 31 03:14:51.663780 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.663741    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:51.663966 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.663813    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:51.663966 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:51.663851    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:51.664081 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.663986    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:51.664081 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664048    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.664030912 +0000 UTC m=+17.222512780 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:51.664518 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664463    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:51.664518 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664482    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:51.664518 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664496    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:51.664713 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664539    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.664525689 +0000 UTC m=+17.223007551 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:51.664713 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664618    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:51.664713 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:51.664655    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:14:59.664643859 +0000 UTC m=+17.223125717 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:52.061178 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:52.061143    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:52.061319 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:52.061280    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:52.061717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:52.061693    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:52.061829 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:52.061803    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:52.061872 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:52.061858    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:52.061960 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:52.061938    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:52.212303 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:52.212266    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7" event={"ID":"13947923-5437-4f19-b97d-c6e913cf8fc7","Type":"ContainerStarted","Data":"cc45451d03e176439a8f96df2b5d4c0811946dd0456fb85befca38e4d5dc3c06"}
Oct 31 03:14:52.996680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:52.996645    2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:14:53.068963 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.068805    2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:14:52.996672967Z","UUID":"a484597a-2544-4b48-a683-40aefece7ca1","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:14:53.078248 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.078219    2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:14:53.078248 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.078257    2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:14:53.230646 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.218367    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7p689" event={"ID":"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6","Type":"ContainerStarted","Data":"2348b12c67797a3edc9bfc614162d6248aa17cf2a0c196870761c2dd405530eb"}
Oct 31 03:14:53.230646 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.222462    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7" event={"ID":"13947923-5437-4f19-b97d-c6e913cf8fc7","Type":"ContainerStarted","Data":"40ff6860b95465723158f1cc2744ecee9f8482ce087b344511cd404277e53fdd"}
Oct 31 03:14:53.236958 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:53.236896    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7p689" podStartSLOduration=2.078946371 podStartE2EDuration="10.236881921s" totalImagesPullingTime="8.15793555s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:53.236406518 +0000 UTC m=+10.794888396" watchObservedRunningTime="2026-10-31 03:14:53.236881921 +0000 UTC m=+10.795363796"
Oct 31 03:14:54.060625 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.060587    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:54.060804 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:54.060737    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:54.061121 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.060945    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:54.061121 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:54.061070    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:54.061263 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.061125    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:54.061263 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:54.061199    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:54.226273 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.226182    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nnxgp" event={"ID":"b1d993de-0f1b-4b88-a998-344eec8c23ac","Type":"ContainerStarted","Data":"9aa7ea5774695ae3e82465a66f2a0cab2e7394f7ebe663a6705a7fcc22e310b1"}
Oct 31 03:14:54.228972 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.228943    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="8afcd5608e097fec33be404d3742a31800fd30299457fe84356a565f8ab3f36e" exitCode=0
Oct 31 03:14:54.229116 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.229054    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"8afcd5608e097fec33be404d3742a31800fd30299457fe84356a565f8ab3f36e"}
Oct 31 03:14:54.244376 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:54.244317    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-nnxgp" podStartSLOduration=2.488678291 podStartE2EDuration="11.244297033s" totalImagesPullingTime="8.755618742s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:54.243743776 +0000 UTC m=+11.802225654" watchObservedRunningTime="2026-10-31 03:14:54.244297033 +0000 UTC m=+11.802778913"
Oct 31 03:14:55.126336 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.126085    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:55.127076 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.127053    2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:14:55.234158 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.234120    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7" event={"ID":"13947923-5437-4f19-b97d-c6e913cf8fc7","Type":"ContainerStarted","Data":"1cb58705ac204760ac9e16729bc3d718ec827957fd27d192d34dead91b2d07dd"}
Oct 31 03:14:55.236548 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.236519    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="48bf176727a45de28bdc3e8519d34fca13be87855d435db5f914062c7319f7da" exitCode=0
Oct 31 03:14:55.236713 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.236617    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"48bf176727a45de28bdc3e8519d34fca13be87855d435db5f914062c7319f7da"}
Oct 31 03:14:55.238957 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.238927    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xrkgp" event={"ID":"c3f060e6-cd83-4284-98ac-d39a6d8a01be","Type":"ContainerStarted","Data":"fd182d3902ee1cd8fa23c68c99038ffb5ff9f92859c53a7567e240112f93a047"}
Oct 31 03:14:55.254720 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.254674    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7" podStartSLOduration=3.467626674 podStartE2EDuration="12.254657483s" totalImagesPullingTime="8.787030809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:55.25394236 +0000 UTC m=+12.812424228" watchObservedRunningTime="2026-10-31 03:14:55.254657483 +0000 UTC m=+12.813139360"
Oct 31 03:14:55.291151 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:55.291063    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xrkgp" podStartSLOduration=3.630012366 podStartE2EDuration="12.291048327s" totalImagesPullingTime="8.661035961s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:55.290771693 +0000 UTC m=+12.849253569" watchObservedRunningTime="2026-10-31 03:14:55.291048327 +0000 UTC m=+12.849530229"
Oct 31 03:14:56.060275 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:56.060234    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:56.060448 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:56.060251    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:56.060448 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:56.060363    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:56.060448 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:56.060432    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:56.060448 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:56.060252    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:56.060692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:56.060531    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:56.242913 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:56.242874    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jmgbj" event={"ID":"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0","Type":"ContainerStarted","Data":"7e36ff84ec974cd789a07f12907317773c428a664fd5cf084415fdb561d81ae1"}
Oct 31 03:14:56.261194 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:56.261142    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jmgbj" podStartSLOduration=2.62840176 podStartE2EDuration="13.261128688s" totalImagesPullingTime="10.632726928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:14:56.260740795 +0000 UTC m=+13.819222684" watchObservedRunningTime="2026-10-31 03:14:56.261128688 +0000 UTC m=+13.819610563"
Oct 31 03:14:57.248292 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:57.248253    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="99de9cc71e1bf533620491cc178b7a8a940df0fcce7740bc49198751d2d6ab47" exitCode=0
Oct 31 03:14:57.248492 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:57.248307    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"99de9cc71e1bf533620491cc178b7a8a940df0fcce7740bc49198751d2d6ab47"}
Oct 31 03:14:58.061216 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:58.061179    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:58.061216 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:58.061197    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:58.061216 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:58.061214    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:58.062068 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:58.061330    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:14:58.062068 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:58.061405    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:14:58.062068 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:58.061486    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:14:59.734167 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:59.733912    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:59.734199    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734077    2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:14:59.734257    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734306    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret podName:c81c0c55-4184-4b1b-a431-268e608e3b60 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.734284149 +0000 UTC m=+33.292766018 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret") pod "global-pull-secret-syncer-bjxm9" (UID: "c81c0c55-4184-4b1b-a431-268e608e3b60") : object "kube-system"/"original-pull-secret" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734360    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734385    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734401    2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734412    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.734397627 +0000 UTC m=+33.292879493 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734416    2573 projected.go:196] Error preparing data for projected volume kube-api-access-98bth for pod openshift-network-diagnostics/network-check-target-4ww6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:14:59.734692 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:14:59.734475    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth podName:fdaeb24f-988d-4df9-93ea-4f9c41a672d0 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.734458977 +0000 UTC m=+33.292940843 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-98bth" (UniqueName: "kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth") pod "network-check-target-4ww6v" (UID: "fdaeb24f-988d-4df9-93ea-4f9c41a672d0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 03:15:00.060819 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:00.060730    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:00.060819 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:00.060757    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:00.060819 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:00.060774    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:00.061046 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:00.060848    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:00.061046 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:00.060888    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:00.061046 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:00.060952    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:00.258233 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:00.258196    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="d1c926738ce57fe16342192d05406a9c1038943ad3ef518164c96b78c75c99ca" exitCode=0
Oct 31 03:15:00.258376 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:00.258237    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"d1c926738ce57fe16342192d05406a9c1038943ad3ef518164c96b78c75c99ca"}
Oct 31 03:15:01.264073 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.263844    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"f0c00a5c68dbb8b9f86f33f8195fee9b83534b52d8be9a94489eea76165d2978"}
Oct 31 03:15:01.264515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.264160    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"8bfa700a1d77b87c5c5b80e9ff42da5ad84fd617a18c5caba895f8781079ce9a"}
Oct 31 03:15:01.264515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.264210    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"b9bdb5cd5c3091edae337a9ee488bc0e7f8a8bd88133c054b63465ced15ab2ce"}
Oct 31 03:15:01.264515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.264239    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"94ecdcc1ec13bda731bd0dd8804b5884f8c2c4671ee847adb30ade6efa63e6ed"}
Oct 31 03:15:01.264515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.264265    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"bc34e9b86b61560c1a0e490c0fc35a812acc74f0a1af3f59b5c80161d4d45814"}
Oct 31 03:15:01.264515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.264307    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"aa0d4b141730271083d4ea09463d9900b1510401d232c2695badd3f53690dc34"}
Oct 31 03:15:01.266939 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.266910    2573 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="29927f1f75f5a7aa60440ae0029e4f214727c5249263effabf7070df9e829d2b" exitCode=0
Oct 31 03:15:01.267030 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:01.266957    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerDied","Data":"29927f1f75f5a7aa60440ae0029e4f214727c5249263effabf7070df9e829d2b"}
Oct 31 03:15:02.060948 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:02.060912    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:02.060948 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:02.060936    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:02.061211 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:02.061088    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:02.061211 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:02.061128    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:02.061211 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:02.061199    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:02.061354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:02.061243    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:02.272361 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:02.272329    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" event={"ID":"85a63323-7c0f-462b-bdf1-f3a532f2d04c","Type":"ContainerStarted","Data":"24bcc59009de89d93364c4def21cbd400876d48df9485d5b9c3162507954e722"}
Oct 31 03:15:02.296761 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:02.296707    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-jjjbw" podStartSLOduration=7.521294973 podStartE2EDuration="19.296693189s" totalImagesPullingTime="11.775398216s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:02.296452921 +0000 UTC m=+19.854934798" watchObservedRunningTime="2026-10-31 03:15:02.296693189 +0000 UTC m=+19.855175065"
Oct 31 03:15:03.277692 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:03.277585    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"45508b4d3ee62582d1af91c7506b2d93aa8d89aaa6e6d79e446430d91f2e807b"}
Oct 31 03:15:04.060556 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:04.060518    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:04.060556 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:04.060548    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:04.060556 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:04.060562    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:04.060802 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:04.060666    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:04.060802 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:04.060747    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:04.060866 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:04.060801    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:05.286090 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:05.286058    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" event={"ID":"478f1160-d546-4149-aee6-7a3f6f3f520f","Type":"ContainerStarted","Data":"308f181898bafdbce8e08475ac079d306872c4cd3b5172ba0cfc632d3874a49c"}
Oct 31 03:15:05.286972 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:05.286445    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:05.301681 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:05.301654    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:05.317096 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:05.317037    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn" podStartSLOduration=6.146675127 podStartE2EDuration="22.317025883s" totalImagesPullingTime="16.170350756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:05.316980285 +0000 UTC m=+22.875462161" watchObservedRunningTime="2026-10-31 03:15:05.317025883 +0000 UTC m=+22.875507759"
Oct 31 03:15:06.061059 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.061022    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:06.061210 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.061117    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:06.061210 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.061128    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:06.061210 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.061203    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:06.061388 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.061235    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:06.061388 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.061350    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:06.289204 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.289168    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:06.289694 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.289242    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:06.306902 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.306872    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:06.969263 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.969233    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjxm9"]
Oct 31 03:15:06.969414 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.969379    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:06.969477 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.969460    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:06.973122 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.973087    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4ww6v"]
Oct 31 03:15:06.973292 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.973275    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:06.973431 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.973404    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:06.973897 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.973876    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bmxnp"]
Oct 31 03:15:06.974004 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:06.973989    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:06.974120 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:06.974100    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:09.060759 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:09.060726    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:09.060759 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:09.060739    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:09.060759 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:09.060745    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:09.061207 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:09.060813    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:09.061207 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:09.060895    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:09.061207 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:09.060959    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:11.060769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:11.060558    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:11.061360 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:11.060582    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:11.061360 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:11.060867    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:11.061360 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:11.060658    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:11.061360 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:11.060930    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:11.061360 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:11.061001    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:12.773494 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:12.773456    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:12.774024 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:12.773594    2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:15:12.774157 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:12.774141    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:13.061706 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.061589    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:13.061706 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:13.061702    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-4ww6v" podUID="fdaeb24f-988d-4df9-93ea-4f9c41a672d0"
Oct 31 03:15:13.061906 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.061720    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:13.061906 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.061746    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:13.061906 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:13.061842    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bjxm9" podUID="c81c0c55-4184-4b1b-a431-268e608e3b60"
Oct 31 03:15:13.062064 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:13.061938    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bmxnp" podUID="40423a9f-278f-4f36-87b1-2d05afa56fa7"
Oct 31 03:15:13.731636 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.731537    2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeReady"
Oct 31 03:15:13.731789 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.731676    2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 03:15:13.766649 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.766599    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dkwqg"]
Oct 31 03:15:13.817746 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.817706    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5f46r"]
Oct 31 03:15:13.818214 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.817844    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:13.821945 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.821425    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 03:15:13.821945 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.821441    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 03:15:13.826002 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.825980    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmx67"]
Oct 31 03:15:13.826141 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.826109    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:13.844337 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.844313    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"]
Oct 31 03:15:13.844470 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.844417    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:13.860293 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.860269    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j2gqr"]
Oct 31 03:15:13.860404 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.860384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:13.863793 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.863764    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 03:15:13.876768 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.876744    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"]
Oct 31 03:15:13.876914 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.876880    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.911317 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.911289    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5497575d9c-lm8wr"]
Oct 31 03:15:13.911444 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.911409    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:13.921488 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921453    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 03:15:13.921488 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921466    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921535    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921540    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921560    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921583    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921632    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921642    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:13.921680 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.921673    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:15:13.924436 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924418    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"]
Oct 31 03:15:13.924526 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924436    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.924526 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924466    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.924526 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924492    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:13.924655 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924542    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:13.924655 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924559    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.924655 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924585    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.924799 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924660    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m979\" (UniqueName: \"kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:13.924799 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924723    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf5nw\" (UniqueName: \"kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw\") pod \"kas-connection-checker-685895c7c9-dkwqg\" (UID: \"e077c72b-608e-43e2-b55f-db29e830b507\") " pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:13.924799 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924756    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lg72n\" (UniqueName: \"kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n\") pod \"kas-connection-checker-685895c7c9-jmx67\" (UID: \"c89742f3-947b-45c2-a14b-491054ba970f\") " pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:13.925057 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924796    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mshws\" (UniqueName: \"kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.925057 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924893    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7x7d\" (UniqueName: \"kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d\") pod \"kas-connection-checker-685895c7c9-5f46r\" (UID: \"0736e363-b4be-4a6f-8fe5-e9a1aeea0222\") " pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:13.925057 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924929    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:13.925057 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.924957    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:13.929652 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.929630    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 03:15:13.929817 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.929773    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 03:15:13.949222 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.949195    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"]
Oct 31 03:15:13.949356 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.949295    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:13.965336 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965309    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 03:15:13.965336 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965315    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 03:15:13.965551 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965340    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 03:15:13.965551 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965399    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 03:15:13.965841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965728    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 03:15:13.965841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965762    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z4k9w\""
Oct 31 03:15:13.965841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965778    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"]
Oct 31 03:15:13.965841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.965830    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gz7pd\""
Oct 31 03:15:13.966077 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.966010    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:13.969598 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.969577    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"]
Oct 31 03:15:13.969731 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.969701    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:13.976284 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.976263    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:15:13.976397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.976384    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:13.986242 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.986223    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"]
Oct 31 03:15:13.986375 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:13.986361    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.001208 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.001188    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:15:14.001326 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.001294    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:14.015316 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.015294    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4927g"]
Oct 31 03:15:14.015456 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.015418    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.022237 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022209    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022237 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022232    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022249    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022257    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022237    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022213    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022317    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022424 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022332    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022432    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022444    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022478    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022510    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022513    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022557    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.022771 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.022597    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 03:15:14.025316 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025294    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8fs\" (UniqueName: \"kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.025431 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025325    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8w7\" (UniqueName: \"kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.025431 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025347    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5nw\" (UniqueName: \"kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw\") pod \"kas-connection-checker-685895c7c9-dkwqg\" (UID: \"e077c72b-608e-43e2-b55f-db29e830b507\") " pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:14.025539 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025427    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7x7d\" (UniqueName: \"kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d\") pod \"kas-connection-checker-685895c7c9-5f46r\" (UID: \"0736e363-b4be-4a6f-8fe5-e9a1aeea0222\") " pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:14.025539 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025477    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg72n\" (UniqueName: \"kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n\") pod \"kas-connection-checker-685895c7c9-jmx67\" (UID: \"c89742f3-947b-45c2-a14b-491054ba970f\") " pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:14.025539 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025511    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.025691 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025536    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.025691 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025633    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.025691 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025675    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.025821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025722    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.025821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025745    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.025821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025775    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.025926 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025879    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.025926 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025911    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.026014 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025941    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.026014 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.025989    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp8wd\" (UniqueName: \"kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.026107 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026014    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m979\" (UniqueName: \"kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.026107 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026033    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026189 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026119    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.026189 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026143    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026189 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026170    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026335 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026193    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.026335 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026221    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.026335 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026262    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026335 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026304    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.026335 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026327    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026360    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5km2s\" (UniqueName: \"kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s\") pod \"volume-data-source-validator-58c9f5f46d-ktx5k\" (UID: \"97e7fbff-1276-4704-9580-c2a43dc17a97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026393    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnsfq\" (UniqueName: \"kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026417    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshws\" (UniqueName: \"kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026456    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026474    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.026536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026501    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026543    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026639    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026665    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026681    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026682    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026777    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.026822 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026792    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 03:15:14.027015 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026946    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.027015 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026964    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl7ss\" (UniqueName: \"kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.027015 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.026983    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.027486 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.027458    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.027575 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.027499    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.027686 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.027667    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.029714 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029699    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmx67"]
Oct 31 03:15:14.029780 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029727    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dkwqg"]
Oct 31 03:15:14.029780 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029744    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5f46r"]
Oct 31 03:15:14.029780 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029760    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j2gqr"]
Oct 31 03:15:14.029780 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029774    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"]
Oct 31 03:15:14.029953 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029788    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ft2z2"]
Oct 31 03:15:14.029953 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.029926    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.030864 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.030843    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.030942 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.030868    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.035320 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.035045    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lbdjh"]
Oct 31 03:15:14.035320 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.035122    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.037485 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.037458    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m979\" (UniqueName: \"kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.037954 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.037932    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7x7d\" (UniqueName: \"kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d\") pod \"kas-connection-checker-685895c7c9-5f46r\" (UID: \"0736e363-b4be-4a6f-8fe5-e9a1aeea0222\") " pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:14.038103 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038083    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshws\" (UniqueName: \"kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.038564 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038544    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"]
Oct 31 03:15:14.038684 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038576    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4927g"]
Oct 31 03:15:14.038684 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038592    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038687    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg72n\" (UniqueName: \"kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n\") pod \"kas-connection-checker-685895c7c9-jmx67\" (UID: \"c89742f3-947b-45c2-a14b-491054ba970f\") " pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038706    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038726    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038739    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038752    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038772    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"]
Oct 31 03:15:14.038782 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038771    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5nw\" (UniqueName: \"kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw\") pod \"kas-connection-checker-685895c7c9-dkwqg\" (UID: \"e077c72b-608e-43e2-b55f-db29e830b507\") " pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:14.039177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038797    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbdjh"]
Oct 31 03:15:14.039177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038818    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ft2z2"]
Oct 31 03:15:14.039177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038830    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5497575d9c-lm8wr"]
Oct 31 03:15:14.039177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038843    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:15:14.039177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.038870    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.065372 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065342    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 03:15:14.065530 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065419    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:15:14.065530 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065430    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 03:15:14.065530 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065442    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 03:15:14.065530 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065490    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065533    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065543    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065568    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065677    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2x9p7\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065720    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065721    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9h5qx\""
Oct 31 03:15:14.065758 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.065735    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mv56x\""
Oct 31 03:15:14.066719 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.066360    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 03:15:14.066719 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.066389    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7fps\""
Oct 31 03:15:14.066719 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.066434    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hcpdk\""
Oct 31 03:15:14.066719 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.066442    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\""
Oct 31 03:15:14.123320 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.123287    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 03:15:14.123512 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.123382    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:15:14.123569 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.123524    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:15:14.123569 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.123549    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:15:14.127939 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.127917    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.127996 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.127955    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.127996 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.127973    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.128051 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128034    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.128083 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128056    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.128083 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128074    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.128140 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128097    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128171 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128147    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.128218 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.128364 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128254    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.62823241 +0000 UTC m=+32.186714287 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:14.128364 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128279    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.128364 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128321    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6lc8\" (UniqueName: \"kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128439    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128477    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128492    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128541    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.128555 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128557    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.628539473 +0000 UTC m=+32.187021340 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128568    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128589    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128637    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8wd\" (UniqueName: \"kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128668    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128694    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128708    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128722    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128758    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128791    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-226x7\" (UniqueName: \"kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7\") pod \"network-check-source-6f78d59f99-29k8x\" (UID: \"4d97b9da-e0f2-418d-bec3-7ebb1a897a47\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128821    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128845    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.128882 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128870    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wzxh\" (UniqueName: \"kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128765    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.128947    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.128969    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls podName:231e395b-1984-48a0-865b-b9b05c489eec nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.628954477 +0000 UTC m=+32.187436341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f49fv" (UID: "231e395b-1984-48a0-865b-b9b05c489eec") : secret "samples-operator-tls" not found
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129025    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129064    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.129506 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129093    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.129817 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129636    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.129874 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129846    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.129944 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129924    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km2s\" (UniqueName: \"kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s\") pod \"volume-data-source-validator-58c9f5f46d-ktx5k\" (UID: \"97e7fbff-1276-4704-9580-c2a43dc17a97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:14.130010 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129958    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.130010 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129987    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsfq\" (UniqueName: \"kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.130010 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.129993    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.130152 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130017    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.130152 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130091    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.130355 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130332    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.130433 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130370    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.130433 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130417    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.130536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130503    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.130591 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130533    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.130591 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130566    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.130591 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.130578    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130592    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130642    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.130652    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.630634932 +0000 UTC m=+32.189116799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : secret "router-metrics-certs-default" not found
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130696    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7ss\" (UniqueName: \"kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130730    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.130769 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130757    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8fs\" (UniqueName: \"kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.131094 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130788    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8w7\" (UniqueName: \"kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.131094 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.130826    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.131094 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.130955    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:14.131094 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.130998    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls podName:378f839e-767e-421d-b8c7-65369e94a937 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.630985604 +0000 UTC m=+32.189467458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2cg4n" (UID: "378f839e-767e-421d-b8c7-65369e94a937") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:14.131373 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.131350    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.131440 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.131376    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.131440 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.131421    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.131553 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.131493    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.131879 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.131860    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.132787 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.132763    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.133241 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.133225    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.140708 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.140668    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.143416 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.143115    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsfq\" (UniqueName: \"kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.143416 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.143399    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8w7\" (UniqueName: \"kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.143734 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.143706    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km2s\" (UniqueName: \"kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s\") pod \"volume-data-source-validator-58c9f5f46d-ktx5k\" (UID: \"97e7fbff-1276-4704-9580-c2a43dc17a97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:14.144034 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.144016    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.144763 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.144742    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8fs\" (UniqueName: \"kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.144966 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.144950    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8wd\" (UniqueName: \"kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.145091 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.145063    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7ss\" (UniqueName: \"kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.155225 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.155204    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:14.168502 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.168481    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:14.180932 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.180912    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:14.202204 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.202177    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:14.218819 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.218781    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:14.231642 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231598    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.231757 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231653    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.231757 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231680    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.231757 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231739    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.231884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231758    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.231884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231786    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.231884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231822    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.231884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lc8\" (UniqueName: \"kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.232006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231882    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.232006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231927    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.232006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231944    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226x7\" (UniqueName: \"kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7\") pod \"network-check-source-6f78d59f99-29k8x\" (UID: \"4d97b9da-e0f2-418d-bec3-7ebb1a897a47\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:14.232006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231966    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzxh\" (UniqueName: \"kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.232006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.231995    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.232154 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.232015    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.232154 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.232034    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.232154 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.232050    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.232154 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.232090    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.232655 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.232588    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:14.232655 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.232627    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:14.232769 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.232697    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.732676203 +0000 UTC m=+32.291158070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:14.232878 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.232859    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:15:14.232955 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.232916    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert podName:9a4b3f3c-5b83-481b-9038-aad4508357bf nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.732901688 +0000 UTC m=+32.291383542 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4927g" (UID: "9a4b3f3c-5b83-481b-9038-aad4508357bf") : secret "networking-console-plugin-cert" not found
Oct 31 03:15:14.233167 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.233149    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:14.233262 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.233204    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:14.733188913 +0000 UTC m=+32.291670785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:14.239887 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.239808    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:14.242285 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.241879    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.243515 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.242396    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.244390 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244165    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.244390 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244165    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.244390 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244316    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.244516 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244428    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.244592 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244561    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.244592 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.244578    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.246830 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.246000    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.246830 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.246402    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lc8\" (UniqueName: \"kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.248821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.247045    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.248821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.247528    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzxh\" (UniqueName: \"kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.248821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.248182    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226x7\" (UniqueName: \"kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7\") pod \"network-check-source-6f78d59f99-29k8x\" (UID: \"4d97b9da-e0f2-418d-bec3-7ebb1a897a47\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:14.248821 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.248287    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.267162 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.267131    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:14.326355 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.326102    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:14.334904 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.334865    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:14.416825 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.416795    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:14.515678 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.515649    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-5f46r"]
Oct 31 03:15:14.518127 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.518098    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-dkwqg"]
Oct 31 03:15:14.519107 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.519085    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-jmx67"]
Oct 31 03:15:14.529455 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.529417    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc89742f3_947b_45c2_a14b_491054ba970f.slice/crio-8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e WatchSource:0}: Error finding container 8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e: Status 404 returned error can't find the container with id 8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e
Oct 31 03:15:14.530221 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.530193    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0736e363_b4be_4a6f_8fe5_e9a1aeea0222.slice/crio-231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039 WatchSource:0}: Error finding container 231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039: Status 404 returned error can't find the container with id 231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039
Oct 31 03:15:14.530801 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.530774    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode077c72b_608e_43e2_b55f_db29e830b507.slice/crio-4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe WatchSource:0}: Error finding container 4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe: Status 404 returned error can't find the container with id 4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe
Oct 31 03:15:14.548579 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.548550    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"]
Oct 31 03:15:14.552039 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.552012    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4f9f2d_e0c9_46f2_8100_06f434f66322.slice/crio-3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3 WatchSource:0}: Error finding container 3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3: Status 404 returned error can't find the container with id 3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3
Oct 31 03:15:14.586174 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.586138    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-j2gqr"]
Oct 31 03:15:14.590927 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.590872    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc41b4e8_af8b_420c_960d_8f948efbf80a.slice/crio-1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336 WatchSource:0}: Error finding container 1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336: Status 404 returned error can't find the container with id 1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336
Oct 31 03:15:14.610823 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.610784    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"]
Oct 31 03:15:14.615893 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.615860    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e7fbff_1276_4704_9580_c2a43dc17a97.slice/crio-14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be WatchSource:0}: Error finding container 14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be: Status 404 returned error can't find the container with id 14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be
Oct 31 03:15:14.625166 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.625139    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"]
Oct 31 03:15:14.629340 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.629309    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b4b412a_ef91_4c3e_89c3_9287445d404c.slice/crio-606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af WatchSource:0}: Error finding container 606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af: Status 404 returned error can't find the container with id 606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af
Oct 31 03:15:14.635873 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.635844    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.635975 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.635937    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.635976    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.636069    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.636101    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636188    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636202    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:14.636334 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636259    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.63624187 +0000 UTC m=+33.194723739 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:14.636677 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636373    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:14.636677 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636423    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls podName:378f839e-767e-421d-b8c7-65369e94a937 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.636407745 +0000 UTC m=+33.194889603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2cg4n" (UID: "378f839e-767e-421d-b8c7-65369e94a937") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:14.636677 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636501    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.636488118 +0000 UTC m=+33.194969986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:14.636677 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636587    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:15:14.636677 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636641    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls podName:231e395b-1984-48a0-865b-b9b05c489eec nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.636629446 +0000 UTC m=+33.195111314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f49fv" (UID: "231e395b-1984-48a0-865b-b9b05c489eec") : secret "samples-operator-tls" not found
Oct 31 03:15:14.636879 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636738    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:15:14.636879 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.636790    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.63677349 +0000 UTC m=+33.195255347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : secret "router-metrics-certs-default" not found
Oct 31 03:15:14.666736 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.666663    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"]
Oct 31 03:15:14.667069 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.667050    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"]
Oct 31 03:15:14.731025 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.730997    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ft2z2"]
Oct 31 03:15:14.733096 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:14.733070    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7d30b6_5160_4f1b_a67b_a0c92f02e0fa.slice/crio-6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086 WatchSource:0}: Error finding container 6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086: Status 404 returned error can't find the container with id 6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086
Oct 31 03:15:14.737177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.737153    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:14.737291 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.737266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:14.737349 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737315    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:14.737349 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:14.737318    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:14.737446 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737386    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.737365427 +0000 UTC m=+33.295847296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:14.737492 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737451    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:14.737492 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737467    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:14.737554 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737492    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:15:14.737554 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737504    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.737490961 +0000 UTC m=+33.295972816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:14.737554 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:14.737544    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert podName:9a4b3f3c-5b83-481b-9038-aad4508357bf nodeName:}" failed. No retries permitted until 2026-10-31 03:15:15.73752841 +0000 UTC m=+33.296010279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4927g" (UID: "9a4b3f3c-5b83-481b-9038-aad4508357bf") : secret "networking-console-plugin-cert" not found
Oct 31 03:15:15.061122 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.061086    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:15.061544 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.061066    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:15.061544 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.061255    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:15.064804 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.064779    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:15:15.064981 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.064955    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\""
Oct 31 03:15:15.066122 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.066094    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\""
Oct 31 03:15:15.066266 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.066241    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:15:15.066266 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.066252    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\""
Oct 31 03:15:15.312414 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.312321    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f" event={"ID":"61a64191-2d4c-48e4-bf08-e010b704cda5","Type":"ContainerStarted","Data":"801f4d629f6f4621f64db5ec11111ee148d0c08a371eddaba45859b30064e7b9"}
Oct 31 03:15:15.314365 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.314290    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" event={"ID":"1b4b412a-ef91-4c3e-89c3-9287445d404c","Type":"ContainerStarted","Data":"606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af"}
Oct 31 03:15:15.318098 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.318061    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dkwqg" event={"ID":"e077c72b-608e-43e2-b55f-db29e830b507","Type":"ContainerStarted","Data":"6a0624e30f093deefbb69143435bd18a2428ad5c71c49c10233878fccd209612"}
Oct 31 03:15:15.318226 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.318120    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-dkwqg" event={"ID":"e077c72b-608e-43e2-b55f-db29e830b507","Type":"ContainerStarted","Data":"4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe"}
Oct 31 03:15:15.320941 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.320871    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x" event={"ID":"4d97b9da-e0f2-418d-bec3-7ebb1a897a47","Type":"ContainerStarted","Data":"3ff2b7cb7e0a14af04524c4b6f436e7c1c6e7eade3a08ed0c3edda11484d5934"}
Oct 31 03:15:15.322717 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.322686    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k" event={"ID":"97e7fbff-1276-4704-9580-c2a43dc17a97","Type":"ContainerStarted","Data":"14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be"}
Oct 31 03:15:15.324364 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.324337    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j2gqr" event={"ID":"bc41b4e8-af8b-420c-960d-8f948efbf80a","Type":"ContainerStarted","Data":"1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336"}
Oct 31 03:15:15.327116 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.327093    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5f46r" event={"ID":"0736e363-b4be-4a6f-8fe5-e9a1aeea0222","Type":"ContainerStarted","Data":"4e8f7aa719f7e9ba8a75533203d1f3714300d4fea8ca6ff7809081475c5d0826"}
Oct 31 03:15:15.327233 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.327137    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-5f46r" event={"ID":"0736e363-b4be-4a6f-8fe5-e9a1aeea0222","Type":"ContainerStarted","Data":"231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039"}
Oct 31 03:15:15.330506 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.330452    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms" event={"ID":"aa4f9f2d-e0c9-46f2-8100-06f434f66322","Type":"ContainerStarted","Data":"3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3"}
Oct 31 03:15:15.333026 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.332994    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ft2z2" event={"ID":"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa","Type":"ContainerStarted","Data":"6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086"}
Oct 31 03:15:15.336659 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.336595    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-dkwqg" podStartSLOduration=407.336584307 podStartE2EDuration="6m47.336584307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:15.335203813 +0000 UTC m=+32.893685690" watchObservedRunningTime="2026-10-31 03:15:15.336584307 +0000 UTC m=+32.895066216"
Oct 31 03:15:15.339060 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.339028    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmx67" event={"ID":"c89742f3-947b-45c2-a14b-491054ba970f","Type":"ContainerStarted","Data":"db471d6644d4d9faa131d446dde41548e421806390f7bb797e2c831a748653bc"}
Oct 31 03:15:15.339153 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.339075    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-jmx67" event={"ID":"c89742f3-947b-45c2-a14b-491054ba970f","Type":"ContainerStarted","Data":"8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e"}
Oct 31 03:15:15.353040 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.352829    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-5f46r" podStartSLOduration=407.352810266 podStartE2EDuration="6m47.352810266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:15.352798152 +0000 UTC m=+32.911280029" watchObservedRunningTime="2026-10-31 03:15:15.352810266 +0000 UTC m=+32.911292124"
Oct 31 03:15:15.371787 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.371015    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-jmx67" podStartSLOduration=407.370994949 podStartE2EDuration="6m47.370994949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:15.37027316 +0000 UTC m=+32.928755039" watchObservedRunningTime="2026-10-31 03:15:15.370994949 +0000 UTC m=+32.929476826"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.647320    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.647444    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.647483    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.647565    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.647651    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.647813    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.647830    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.647890    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.647871666 +0000 UTC m=+35.206353525 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648344    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648395    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls podName:231e395b-1984-48a0-865b-b9b05c489eec nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.648381183 +0000 UTC m=+35.206863043 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f49fv" (UID: "231e395b-1984-48a0-865b-b9b05c489eec") : secret "samples-operator-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648453    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648487    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.648476242 +0000 UTC m=+35.206958098 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : secret "router-metrics-certs-default" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648537    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648569    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls podName:378f839e-767e-421d-b8c7-65369e94a937 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.648559316 +0000 UTC m=+35.207041172 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2cg4n" (UID: "378f839e-767e-421d-b8c7-65369e94a937") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:15.648742 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.648651    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.648641211 +0000 UTC m=+35.207123066 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748354    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748460    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748529    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748577    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748678    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.748711    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.748891    2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.748954    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs podName:40423a9f-278f-4f36-87b1-2d05afa56fa7 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:47.748936926 +0000 UTC m=+65.307418782 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs") pod "network-metrics-daemon-bmxnp" (UID: "40423a9f-278f-4f36-87b1-2d05afa56fa7") : secret "metrics-daemon-secret" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749259    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749275    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749322    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.749306849 +0000 UTC m=+35.307788709 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749386    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749416    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.749405825 +0000 UTC m=+35.307887683 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.749947    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:15:15.750354 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:15.750001    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert podName:9a4b3f3c-5b83-481b-9038-aad4508357bf nodeName:}" failed. No retries permitted until 2026-10-31 03:15:17.74998524 +0000 UTC m=+35.308467097 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4927g" (UID: "9a4b3f3c-5b83-481b-9038-aad4508357bf") : secret "networking-console-plugin-cert" not found
Oct 31 03:15:15.756563 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.756466    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:15.758373 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:15.758298    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:16.031031 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:16.030948    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:16.051884 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:16.051246    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:16.371872 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:16.371425    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-4ww6v"]
Oct 31 03:15:16.374834 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:16.374795    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdaeb24f_988d_4df9_93ea_4f9c41a672d0.slice/crio-6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626 WatchSource:0}: Error finding container 6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626: Status 404 returned error can't find the container with id 6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626
Oct 31 03:15:16.411571 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:16.411482    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bjxm9"]
Oct 31 03:15:16.428712 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:16.428629    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc81c0c55_4184_4b1b_a431_268e608e3b60.slice/crio-2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220 WatchSource:0}: Error finding container 2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220: Status 404 returned error can't find the container with id 2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220
Oct 31 03:15:17.353897 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.353774    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4ww6v" event={"ID":"fdaeb24f-988d-4df9-93ea-4f9c41a672d0","Type":"ContainerStarted","Data":"6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626"}
Oct 31 03:15:17.357706 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.356842    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k" event={"ID":"97e7fbff-1276-4704-9580-c2a43dc17a97","Type":"ContainerStarted","Data":"7f56a999a51d0bec14c1fde2205ddd1863c873f76e5a5b85a156aedc77a304dc"}
Oct 31 03:15:17.359234 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.359202    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bjxm9" event={"ID":"c81c0c55-4184-4b1b-a431-268e608e3b60","Type":"ContainerStarted","Data":"2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220"}
Oct 31 03:15:17.374403 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.374343    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k" podStartSLOduration=375.860633091 podStartE2EDuration="6m18.37432422s" totalImagesPullingTime="2.513691129s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:17.373243204 +0000 UTC m=+34.931725081" watchObservedRunningTime="2026-10-31 03:15:17.37432422 +0000 UTC m=+34.932806097"
Oct 31 03:15:17.683702 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.683660    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:17.683871 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.683727    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:17.683871 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.683791    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:17.683871 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.683823    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:17.683871 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.683855    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:17.684018 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684005    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:15:17.684067 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684057    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls podName:231e395b-1984-48a0-865b-b9b05c489eec nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.684044797 +0000 UTC m=+39.242526652 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f49fv" (UID: "231e395b-1984-48a0-865b-b9b05c489eec") : secret "samples-operator-tls" not found
Oct 31 03:15:17.684489 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684461    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:15:17.684694 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684523    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.684506715 +0000 UTC m=+39.242988587 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : secret "router-metrics-certs-default" not found
Oct 31 03:15:17.684694 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684579    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:17.684694 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684628    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls podName:378f839e-767e-421d-b8c7-65369e94a937 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.684600938 +0000 UTC m=+39.243082795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2cg4n" (UID: "378f839e-767e-421d-b8c7-65369e94a937") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:17.684694 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684690    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.684681071 +0000 UTC m=+39.243162926 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:17.684979 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684747    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:17.684979 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684758    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:17.684979 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.684786    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.684777617 +0000 UTC m=+39.243259488 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.784908    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.785048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:17.785103    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785236    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785252    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785303    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.785285772 +0000 UTC m=+39.343767627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785671    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785712    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.785701238 +0000 UTC m=+39.344183104 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785750    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:15:17.785855 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:17.785771    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert podName:9a4b3f3c-5b83-481b-9038-aad4508357bf nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.785765103 +0000 UTC m=+39.344246957 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4927g" (UID: "9a4b3f3c-5b83-481b-9038-aad4508357bf") : secret "networking-console-plugin-cert" not found
Oct 31 03:15:20.383477 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.383446    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/0.log"
Oct 31 03:15:20.383869 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.383492    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerID="f37a1293afc24da77c566315ffccf15cd9f13ec765434e1c465f25ec0d328c48" exitCode=255
Oct 31 03:15:20.383869 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.383577    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" event={"ID":"1b4b412a-ef91-4c3e-89c3-9287445d404c","Type":"ContainerDied","Data":"f37a1293afc24da77c566315ffccf15cd9f13ec765434e1c465f25ec0d328c48"}
Oct 31 03:15:20.384000 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.383984    2573 scope.go:122] "RemoveContainer" containerID="f37a1293afc24da77c566315ffccf15cd9f13ec765434e1c465f25ec0d328c48"
Oct 31 03:15:20.386702 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.386673    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j2gqr" event={"ID":"bc41b4e8-af8b-420c-960d-8f948efbf80a","Type":"ContainerStarted","Data":"d71266a14902c3149842c1e03b7407d414e2cf93dfc32167b15f428329d9a5cd"}
Oct 31 03:15:20.390227 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.390193    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms" event={"ID":"aa4f9f2d-e0c9-46f2-8100-06f434f66322","Type":"ContainerStarted","Data":"370e937027cfa282cca915ad1f8231a69a10f117ae89459f5430f5c3acf3bbde"}
Oct 31 03:15:20.456151 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.455298    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms" podStartSLOduration=402.062150283 podStartE2EDuration="6m47.455275924s" totalImagesPullingTime="5.393125641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:20.422408937 +0000 UTC m=+37.980890815" watchObservedRunningTime="2026-10-31 03:15:20.455275924 +0000 UTC m=+38.013757806"
Oct 31 03:15:20.939875 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.939775    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-j2gqr" podStartSLOduration=401.575944504 podStartE2EDuration="6m46.939757439s" totalImagesPullingTime="5.363812935s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:20.456590755 +0000 UTC m=+38.015072639" watchObservedRunningTime="2026-10-31 03:15:20.939757439 +0000 UTC m=+38.498239332"
Oct 31 03:15:20.941014 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.940984    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4crls"]
Oct 31 03:15:20.952520 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.952487    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:15:20.953630 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.952820    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:20.958730 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.958452    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:15:20.958730 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.958520    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:15:20.958730 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.958684    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\""
Oct 31 03:15:20.975143 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.974977    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4crls"]
Oct 31 03:15:20.975143 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.975038    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:15:20.975143 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:20.975139    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.024325 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.023817    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rrfxr\""
Oct 31 03:15:21.082668 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.082010    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"]
Oct 31 03:15:21.094723 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.094691    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"]
Oct 31 03:15:21.095006 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.094983    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:21.112200 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.112158    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4hz9x\""
Oct 31 03:15:21.120883 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.120854    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"]
Oct 31 03:15:21.121016 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.120904    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"]
Oct 31 03:15:21.121016 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.121006    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:21.132123 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132089    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.132272 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132150    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.132272 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132179    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.132272 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132239    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgvz8\" (UniqueName: \"kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.132395 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132271    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.132395 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132331    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.132395 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132381    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.132477 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132405    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.132477 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.132437    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.197683 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.197563    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 03:15:21.198079 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.197709    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.233885    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvz8\" (UniqueName: \"kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.233935    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spvqm\" (UniqueName: \"kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm\") pod \"migrator-66df5d496b-xkkh4\" (UID: \"290c845c-1024-4d81-a60f-7feb2a3d975a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.233962    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234010    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234046    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9lq\" (UniqueName: \"kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq\") pod \"migrator-66df5d496b-2wr8b\" (UID: \"29ff3f9f-68da-43f9-a824-b759e77376e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234063    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234078    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234098    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234191    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234220    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234234    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.234387    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.235020    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.235108    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.235386    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.235356    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:21.235496 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.235460    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.236098 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.235445    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.735431677 +0000 UTC m=+39.293913531 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : configmap "kube-rbac-proxy" not found
Oct 31 03:15:21.236098 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.235533    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:21.735505918 +0000 UTC m=+39.293987788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:21.236098 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.235801    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.248308 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.248282    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.248459 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.248334    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvz8\" (UniqueName: \"kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.335079 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.335048    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9lq\" (UniqueName: \"kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq\") pod \"migrator-66df5d496b-2wr8b\" (UID: \"29ff3f9f-68da-43f9-a824-b759e77376e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:21.335262 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.335248    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvqm\" (UniqueName: \"kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm\") pod \"migrator-66df5d496b-xkkh4\" (UID: \"290c845c-1024-4d81-a60f-7feb2a3d975a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:21.344085 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.343994    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9lq\" (UniqueName: \"kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq\") pod \"migrator-66df5d496b-2wr8b\" (UID: \"29ff3f9f-68da-43f9-a824-b759e77376e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:21.344085 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.344029    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvqm\" (UniqueName: \"kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm\") pod \"migrator-66df5d496b-xkkh4\" (UID: \"290c845c-1024-4d81-a60f-7feb2a3d975a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:21.347844 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.347817    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:21.400383 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.400266    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ft2z2" event={"ID":"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa","Type":"ContainerStarted","Data":"7ffeaa3f83672be234d48fbc8b286338d9a8250ca79abe2b8276b3e4e3199afe"}
Oct 31 03:15:21.403590 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.402508    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f" event={"ID":"61a64191-2d4c-48e4-bf08-e010b704cda5","Type":"ContainerStarted","Data":"d255b1d9a28f402d451aaba83f68d0445b5091fcb668a88099d83d0169930526"}
Oct 31 03:15:21.411114 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.410972    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.414331    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/0.log"
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.414364    2573 generic.go:356] "Generic (PLEG): container finished" podID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421" exitCode=255
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.414989    2573 scope.go:122] "RemoveContainer" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.415124    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.415320    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" event={"ID":"1b4b412a-ef91-4c3e-89c3-9287445d404c","Type":"ContainerDied","Data":"660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"}
Oct 31 03:15:21.415640 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.415359    2573 scope.go:122] "RemoveContainer" containerID="f37a1293afc24da77c566315ffccf15cd9f13ec765434e1c465f25ec0d328c48"
Oct 31 03:15:21.422177 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.422135    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ft2z2" podStartSLOduration=1.923019644 podStartE2EDuration="8.422124929s" totalImagesPullingTime="6.499105285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:21.421633422 +0000 UTC m=+38.980115295" watchObservedRunningTime="2026-10-31 03:15:21.422124929 +0000 UTC m=+38.980606805"
Oct 31 03:15:21.438418 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.434727    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:21.467072 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.467046    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:21.589725 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.588510    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f" podStartSLOduration=376.986203959 podStartE2EDuration="6m22.588491543s" totalImagesPullingTime="5.602287584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:21.456723864 +0000 UTC m=+39.015205741" watchObservedRunningTime="2026-10-31 03:15:21.588491543 +0000 UTC m=+39.146973482"
Oct 31 03:15:21.589725 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.588663    2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sqvnc"]
Oct 31 03:15:21.596397 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.596297    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.606494 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.605758    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:15:21.606689 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.606538    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:15:21.606689 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.606587    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:15:21.606823 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.606764    2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:15:21.606880 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.606854    2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4pxs\""
Oct 31 03:15:21.611402 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.608163    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sqvnc"]
Oct 31 03:15:21.663269 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.663171    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:15:21.666224 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:21.666194    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34e0666_5012_40e2_a7b3_7f6e4470545a.slice/crio-ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81 WatchSource:0}: Error finding container ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81: Status 404 returned error can't find the container with id ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81
Oct 31 03:15:21.742007 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.741971    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.742134 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742053    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:21.742134 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742108    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:21.742134 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742132    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.742298 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742152    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:21.742298 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742195    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:21.742298 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742246    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:21.742298 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742266    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:21.742298 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742282    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.742536 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.742300    2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxk99\" (UniqueName: \"kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.742536 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742413    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:15:21.742536 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742469    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:22.742453395 +0000 UTC m=+40.300935249 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : configmap "kube-rbac-proxy" not found
Oct 31 03:15:21.742536 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742517    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.742510672 +0000 UTC m=+47.300992539 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742572    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742580    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742630    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.742598083 +0000 UTC m=+47.301079950 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742700    2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742724    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls podName:231e395b-1984-48a0-865b-b9b05c489eec nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.742715982 +0000 UTC m=+47.301197848 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-f49fv" (UID: "231e395b-1984-48a0-865b-b9b05c489eec") : secret "samples-operator-tls" not found
Oct 31 03:15:21.742777 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742766    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:21.743004 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742787    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:22.742779791 +0000 UTC m=+40.301261658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:21.743004 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742826    2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Oct 31 03:15:21.743004 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742845    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.742838987 +0000 UTC m=+47.301320855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : secret "router-metrics-certs-default" not found
Oct 31 03:15:21.743004 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742883    2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:21.743004 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.742909    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls podName:378f839e-767e-421d-b8c7-65369e94a937 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.742902276 +0000 UTC m=+47.301384130 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-2cg4n" (UID: "378f839e-767e-421d-b8c7-65369e94a937") : secret "cluster-monitoring-operator-tls" not found
Oct 31 03:15:21.800050 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.799985    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"]
Oct 31 03:15:21.805299 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:21.805025    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod290c845c_1024_4d81_a60f_7feb2a3d975a.slice/crio-9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5 WatchSource:0}: Error finding container 9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5: Status 404 returned error can't find the container with id 9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5
Oct 31 03:15:21.821290 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.821228    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"]
Oct 31 03:15:21.824836 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:21.824806    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ff3f9f_68da_43f9_a824_b759e77376e3.slice/crio-8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14 WatchSource:0}: Error finding container 8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14: Status 404 returned error can't find the container with id 8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843142    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843205    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843235    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk99\" (UniqueName: \"kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843263    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843389    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.843422    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:21.843587 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.843555    2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.844619    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert podName:9a4b3f3c-5b83-481b-9038-aad4508357bf nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.844584851 +0000 UTC m=+47.403066704 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-4927g" (UID: "9a4b3f3c-5b83-481b-9038-aad4508357bf") : secret "networking-console-plugin-cert" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.843703    2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.844673    2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.844373    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.844715    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.844703268 +0000 UTC m=+47.403185127 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.844218    2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:21.844786 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:21.844749    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:29.844741421 +0000 UTC m=+47.403223280 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:21.847438 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.847409    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.854778 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.854726    2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk99\" (UniqueName: \"kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:21.920251 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:21.919982    2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:22.403688 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.401420    2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sqvnc"]
Oct 31 03:15:22.404765 ip-10-0-129-132 kubenswrapper[2573]: W1031 03:15:22.404724    2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod859d15d2_a1b9_4c23_b801_8eceb5939a3e.slice/crio-61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df WatchSource:0}: Error finding container 61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df: Status 404 returned error can't find the container with id 61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df
Oct 31 03:15:22.419900 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.419813    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b" event={"ID":"29ff3f9f-68da-43f9-a824-b759e77376e3","Type":"ContainerStarted","Data":"8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14"}
Oct 31 03:15:22.421841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.421713    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-278hz-d9lkp" event={"ID":"b34e0666-5012-40e2-a7b3-7f6e4470545a","Type":"ContainerStarted","Data":"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"}
Oct 31 03:15:22.421841 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.421774    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-278hz-d9lkp" event={"ID":"b34e0666-5012-40e2-a7b3-7f6e4470545a","Type":"ContainerStarted","Data":"ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81"}
Oct 31 03:15:22.423308 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.423278    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4" event={"ID":"290c845c-1024-4d81-a60f-7feb2a3d975a","Type":"ContainerStarted","Data":"9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5"}
Oct 31 03:15:22.425256 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.425232    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:22.425618 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.425591    2573 scope.go:122] "RemoveContainer" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"
Oct 31 03:15:22.425817 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:22.425792    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:22.428161 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.428109    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc" event={"ID":"859d15d2-a1b9-4c23-b801-8eceb5939a3e","Type":"ContainerStarted","Data":"61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df"}
Oct 31 03:15:22.429799 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.429766    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x" event={"ID":"4d97b9da-e0f2-418d-bec3-7ebb1a897a47","Type":"ContainerStarted","Data":"5ff83e9e52f8959587220253324449fb78417ca0b2cfffe48c646df5f4957672"}
Oct 31 03:15:22.442049 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.441996    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-278hz-d9lkp" podStartSLOduration=2.441979028 podStartE2EDuration="2.441979028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:22.440689351 +0000 UTC m=+39.999171228" watchObservedRunningTime="2026-10-31 03:15:22.441979028 +0000 UTC m=+40.000460905"
Oct 31 03:15:22.493252 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.493196    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x" podStartSLOduration=376.033747054 podStartE2EDuration="6m23.493176192s" totalImagesPullingTime="7.459429138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:08:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:22.463527552 +0000 UTC m=+40.022009427" watchObservedRunningTime="2026-10-31 03:15:22.493176192 +0000 UTC m=+40.051658069"
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.753915    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:22.754005    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:22.754123    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:22.754174    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:24.754160687 +0000 UTC m=+42.312642541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : configmap "kube-rbac-proxy" not found
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:22.754503    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:22.754627 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:22.754539    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:24.754529297 +0000 UTC m=+42.313011151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:23.436154 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.436118    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-4ww6v" event={"ID":"fdaeb24f-988d-4df9-93ea-4f9c41a672d0","Type":"ContainerStarted","Data":"eae8e08fcf181fdb5de35e894342c20f215a05bb3b117a1bc32d2507a066c44a"}
Oct 31 03:15:23.436980 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.436908    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:23.439295 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.438958    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc" event={"ID":"859d15d2-a1b9-4c23-b801-8eceb5939a3e","Type":"ContainerStarted","Data":"e4f39fdd50fb41bb17d5e7cc1beb080c51734ff4b724a7503258c7e8cc7311d4"}
Oct 31 03:15:23.459732 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.459659    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-4ww6v" podStartSLOduration=33.919741747 podStartE2EDuration="40.459638158s" totalImagesPullingTime="6.539896411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:14:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:23.454767226 +0000 UTC m=+41.013249126" watchObservedRunningTime="2026-10-31 03:15:23.459638158 +0000 UTC m=+41.018120032"
Oct 31 03:15:23.474468 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.474374    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc" podStartSLOduration=2.4743544650000002 podStartE2EDuration="2.474354465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:23.472558837 +0000 UTC m=+41.031040714" watchObservedRunningTime="2026-10-31 03:15:23.474354465 +0000 UTC m=+41.032836343"
Oct 31 03:15:23.936892 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:23.936862    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnxgp_b1d993de-0f1b-4b88-a998-344eec8c23ac/dns-node-resolver/0.log"
Oct 31 03:15:24.240520 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.240497    2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:24.240645 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.240557    2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:24.241000 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.240970    2573 scope.go:122] "RemoveContainer" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"
Oct 31 03:15:24.241246 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:24.241220    2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:24.445493 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.445453    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4" event={"ID":"290c845c-1024-4d81-a60f-7feb2a3d975a","Type":"ContainerStarted","Data":"d5fbacb482fbadbdca3b1032c13835f33fe78c3d4a0decfa81db2449b1bac348"}
Oct 31 03:15:24.447382 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.447350    2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b" event={"ID":"29ff3f9f-68da-43f9-a824-b759e77376e3","Type":"ContainerStarted","Data":"945530377bb7e0d7dce97ab0bb2df5d9bb97e6c616dc8d45ccc1df97a9b6ffdc"}
Oct 31 03:15:24.464425 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.464366    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4" podStartSLOduration=1.030398 podStartE2EDuration="3.464347754s" totalImagesPullingTime="2.433949754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:24.464055035 +0000 UTC m=+42.022536911" watchObservedRunningTime="2026-10-31 03:15:24.464347754 +0000 UTC m=+42.022829633"
Oct 31 03:15:24.493143 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.493045    2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b" podStartSLOduration=1.083215227 podStartE2EDuration="3.493027289s" totalImagesPullingTime="2.409812062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:24.491416971 +0000 UTC m=+42.049898853" watchObservedRunningTime="2026-10-31 03:15:24.493027289 +0000 UTC m=+42.051509163"
Oct 31 03:15:24.787995 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.787960    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:24.788150 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.788073    2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:24.788150 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:24.788089    2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 03:15:24.788241 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:24.788171    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:28.788152986 +0000 UTC m=+46.346634840 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : configmap "kube-rbac-proxy" not found
Oct 31 03:15:24.788241 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:24.788186    2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:24.788241 ip-10-0-129-132 kubenswrapper[2573]: E1031 03:15:24.788226    2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:28.788215297 +0000 UTC m=+46.346697151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:24.925643 ip-10-0-129-132 kubenswrapper[2573]: I1031 03:15:24.925594    2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wtqp4_79a249fe-49bc-4f12-b1f5-c2e4b3ecc990/node-ca/0.log"
Oct 31 03:15:25.411966 ip-10-0-129-132 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 03:15:25.420312 ip-10-0-129-132 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 03:15:25.420484 ip-10-0-129-132 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 03:15:25.420535 ip-10-0-129-132 systemd[1]: kubelet.service: Consumed 3.307s CPU time.
Oct 31 03:15:25.440185 ip-10-0-129-132 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 03:15:25.518990 ip-10-0-129-132 kubenswrapper[5098]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:15:25.518990 ip-10-0-129-132 kubenswrapper[5098]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 03:15:25.518990 ip-10-0-129-132 kubenswrapper[5098]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:15:25.518990 ip-10-0-129-132 kubenswrapper[5098]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 03:15:25.521164 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521045    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.521164 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521161    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.521164 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521165    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.521164 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521169    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.521164 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521172    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521175    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521178    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521181    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521184    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521187    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521190    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521193    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521196    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521199    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521204    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521208    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521211    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521214    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521217    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521220    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521222    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521225    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521227    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.521364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521236    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521239    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521241    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521244    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521247    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521250    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521253    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521256    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521259    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521262    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521265    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521267    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521270    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521273    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521275    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521278    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521281    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521284    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521286    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521289    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.521867 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521292    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521295    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521297    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521300    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521303    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521306    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521308    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521312    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521315    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521317    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521320    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521322    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521325    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521327    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521330    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521332    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521336    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521339    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521342    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521344    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.522378 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521347    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521350    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521352    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521354    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521358    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521360    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521363    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521366    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521368    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521372    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521374    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521377    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521383    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521386    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521390    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521392    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521395    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521398    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521400    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.522892 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521403    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521406    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521410    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521412    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521415    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521417    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521420    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521422    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521425    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521428    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521433    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521435    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521438    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521441    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521444    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521446    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521449    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521452    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521454    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521457    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.523364 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521459    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521462    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521465    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.521467    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522334    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522344    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522348    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522352    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522355    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522358    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522360    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522363    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522366    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522368    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522371    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522374    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522376    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522379    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522382    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522385    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.523900 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522388    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522391    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522394    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522397    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522400    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522403    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522405    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522408    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522411    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522413    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522416    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522419    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522421    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522424    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522427    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522430    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522432    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522435    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522437    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.524397 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522440    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522443    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522446    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522448    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522451    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522453    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522456    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522458    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522462    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522466    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522468    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522471    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522473    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522476    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522479    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522483    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522488    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522491    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.524942 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522494    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522497    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522500    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522503    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522505    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522508    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522511    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522513    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522516    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522519    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522521    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522524    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522526    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522529    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522531    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522541    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522544    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522547    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522549    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522556    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.525385 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522559    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522561    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522564    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522567    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522569    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522572    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522575    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522577    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522580    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522583    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522586    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522589    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522592    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522595    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522597    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522600    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522617    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522620    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522623    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522625    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.525921 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522628    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522631    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522633    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522636    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522639    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522641    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522644    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522647    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522656    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522659    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522661    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522664    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.522667    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523223    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523228    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523231    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523234    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523237    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523239    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.526518 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523242    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523245    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523251    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523253    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523256    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523259    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523261    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523264    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523266    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523269    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523272    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523276    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523280    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523284    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523287    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523290    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523292    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523295    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523298    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.527107 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523300    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523303    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523306    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523309    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523311    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523314    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523317    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523319    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523322    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523324    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523327    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523329    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523332    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523334    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523337    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523341    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523344    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523346    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523349    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.527772 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523351    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523354    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523356    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523359    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523362    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523364    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523367    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523369    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523373    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523376    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523379    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523382    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523384    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523387    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523390    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523393    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523396    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523399    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523402    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523405    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.528585 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523407    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523410    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523413    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523415    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523418    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523420    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523423    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523425    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523429    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523431    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523434    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523436    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523439    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523441    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523444    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523447    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523449    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523452    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523455    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523457    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.529489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523462    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523464    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523468    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523472    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523475    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523478    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523481    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523483    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523486    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523489    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523491    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523494    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523497    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523500    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523503    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523505    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523508    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523511    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523514    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.530375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523516    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523519    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.523522    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523650    5098 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523660    5098 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523668    5098 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523673    5098 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523678    5098 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523682    5098 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523686    5098 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523691    5098 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523694    5098 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523698    5098 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523703    5098 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523707    5098 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523710    5098 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523714    5098 flags.go:64] FLAG: --cgroup-root=""
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523717    5098 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523720    5098 flags.go:64] FLAG: --client-ca-file=""
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523723    5098 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523734    5098 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523740    5098 flags.go:64] FLAG: --cluster-domain=""
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523743    5098 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523747    5098 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 03:15:25.531234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523750    5098 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523754    5098 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523758    5098 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523762    5098 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523765    5098 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523768    5098 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523771    5098 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523775    5098 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523779    5098 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523782    5098 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523785    5098 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523790    5098 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523793    5098 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523796    5098 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523800    5098 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523803    5098 flags.go:64] FLAG: --enable-server="true"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523806    5098 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523811    5098 flags.go:64] FLAG: --event-burst="100"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523814    5098 flags.go:64] FLAG: --event-qps="50"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523817    5098 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523821    5098 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523825    5098 flags.go:64] FLAG: --eviction-hard=""
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523830    5098 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523833    5098 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523836    5098 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 03:15:25.532288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523840    5098 flags.go:64] FLAG: --eviction-soft=""
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523843    5098 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523846    5098 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523850    5098 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523853    5098 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523856    5098 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523859    5098 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523863    5098 flags.go:64] FLAG: --feature-gates=""
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523867    5098 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523871    5098 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523874    5098 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523877    5098 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523881    5098 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523884    5098 flags.go:64] FLAG: --help="false"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523887    5098 flags.go:64] FLAG: --hostname-override="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523891    5098 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523894    5098 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523896    5098 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523900    5098 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523904    5098 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523907    5098 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523910    5098 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523913    5098 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 03:15:25.534070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523916    5098 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523920    5098 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523923    5098 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523926    5098 flags.go:64] FLAG: --kube-reserved=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523929    5098 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523933    5098 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523937    5098 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523940    5098 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523943    5098 flags.go:64] FLAG: --lock-file=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523946    5098 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523949    5098 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523952    5098 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523958    5098 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523961    5098 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523965    5098 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523969    5098 flags.go:64] FLAG: --logging-format="text"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523972    5098 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523979    5098 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523983    5098 flags.go:64] FLAG: --manifest-url=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523986    5098 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523990    5098 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523994    5098 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.523998    5098 flags.go:64] FLAG: --max-pods="110"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524001    5098 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524004    5098 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 03:15:25.534787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524007    5098 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524010    5098 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524013    5098 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524016    5098 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524021    5098 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524032    5098 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524036    5098 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524041    5098 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524045    5098 flags.go:64] FLAG: --pod-cidr=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524049    5098 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524053    5098 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524057    5098 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524064    5098 flags.go:64] FLAG: --port="10250"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524068    5098 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524072    5098 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-07b853e1a036cbd75"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524075    5098 flags.go:64] FLAG: --qos-reserved=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524078    5098 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524081    5098 flags.go:64] FLAG: --register-node="true"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524084    5098 flags.go:64] FLAG: --register-with-taints=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524089    5098 flags.go:64] FLAG: --registry-burst="10"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524092    5098 flags.go:64] FLAG: --registry-qps="5"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524095    5098 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524099    5098 flags.go:64] FLAG: --reserved-memory=""
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524103    5098 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524107    5098 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524110    5098 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 03:15:25.535533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524114    5098 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524117    5098 flags.go:64] FLAG: --runonce="false"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524120    5098 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524123    5098 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524126    5098 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524129    5098 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524132    5098 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524135    5098 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524138    5098 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524142    5098 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524145    5098 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524148    5098 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524151    5098 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524154    5098 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524157    5098 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524160    5098 flags.go:64] FLAG: --system-cgroups=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524163    5098 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524169    5098 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524174    5098 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524177    5098 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524182    5098 flags.go:64] FLAG: --tls-min-version=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524185    5098 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524187    5098 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524191    5098 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524194    5098 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 03:15:25.536284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524197    5098 flags.go:64] FLAG: --v="2"
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524201    5098 flags.go:64] FLAG: --version="false"
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524205    5098 flags.go:64] FLAG: --vmodule=""
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524211    5098 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524215    5098 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524363    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524368    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524371    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524374    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524377    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524380    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524383    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524386    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524388    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524392    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524396    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524399    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524402    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524405    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524408    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.536963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524411    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524414    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524416    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524419    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524422    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524426    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524429    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524432    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524435    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524438    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524440    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524443    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524445    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524448    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524450    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524454    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524457    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524459    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524462    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.537466 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524465    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524468    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524471    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524474    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524476    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524479    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524482    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524485    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524487    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524490    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524492    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524495    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524498    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524500    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524503    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524505    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524508    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524510    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524514    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.537939 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524517    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524520    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524522    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524525    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524528    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524530    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524533    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524535    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524538    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524542    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524545    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524547    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524550    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524552    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524555    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524558    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524561    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524564    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524566    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524569    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.538405 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524571    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524574    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524576    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524579    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524581    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524584    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524587    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524590    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524592    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524595    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524597    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524601    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524617    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524620    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524623    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524625    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524628    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524630    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524633    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524636    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.539004 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524638    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524643    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524645    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524648    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524651    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524654    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524657    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524659    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524662    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524665    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524668    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524672    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524675    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.524706    5098 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524746    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524750    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.539598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524753    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524756    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524760    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524763    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524766    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524769    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524771    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524775    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524781    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524784    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524788    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524791    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524794    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524796    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524799    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524802    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524804    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524809    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524813    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.540130 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524816    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524819    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524822    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524825    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524827    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524830    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524833    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524836    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524839    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524841    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524844    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524848    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524851    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524853    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524856    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524858    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524861    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524863    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524865    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.540678 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524868    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524871    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524875    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524878    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524881    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524883    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524886    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524889    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524891    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524894    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524896    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524900    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524903    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524906    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524908    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524911    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524913    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524916    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524919    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524921    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.541191 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524924    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524927    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524930    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524932    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524935    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524937    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524940    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524943    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524946    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524949    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524951    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524954    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524957    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524959    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524964    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524966    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524969    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524971    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524975    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524977    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.541753 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524980    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524982    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524985    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524989    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524991    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524994    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524997    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.524999    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525002    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525005    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525007    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525010    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525012    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525015    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525018    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525021    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525024    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525027    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525030    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525032    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.542420 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525035    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525037    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525040    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525042    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525045    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.525047    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.530454    5098 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.530479    5098 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.530506    5098 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.530514    5098 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530625    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530636    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530641    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530646    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530650    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530655    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530660    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530664    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530668    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.543193 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530673    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530677    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530681    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530685    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530690    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530694    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530699    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530703    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530708    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530713    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530717    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530721    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530726    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530730    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530735    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530739    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530744    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530748    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530752    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.543750 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530757    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530771    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530776    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530780    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530785    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530789    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530794    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530798    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530803    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530807    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530811    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530816    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530820    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530824    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530829    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530833    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530837    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530842    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530846    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530850    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.544223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530856    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530860    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530865    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530869    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530874    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530878    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530882    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530887    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530894    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530900    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530905    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530910    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530915    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530919    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530934    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530939    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530944    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530948    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530953    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.544856 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530957    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530961    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530965    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530970    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530974    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530979    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530983    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530988    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530992    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.530996    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531001    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531005    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531009    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531014    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531018    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531023    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531027    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531032    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531037    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531041    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.545340 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531046    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531050    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531054    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531059    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531063    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531068    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531072    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531076    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531088    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531093    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531097    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531104    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531111    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531117    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531122    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531126    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531140    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531145    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.545884 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531149    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.531199    5098 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531501    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531512    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531516    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531521    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531526    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531530    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531535    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531539    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531543    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531548    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531552    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531556    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531560    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531565    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.546338 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531570    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531574    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531579    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531583    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531588    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531592    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531597    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.531690    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532349    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532549    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532557    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532562    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532569    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532573    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532578    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532583    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532587    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532592    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532597    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.546768 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532624    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532631    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532636    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532640    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532645    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532650    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532654    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532658    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532663    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532668    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532689    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532695    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532700    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532709    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532717    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532724    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532748    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532757    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532763    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.547430 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532767    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532772    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532815    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532820    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532825    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532829    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532841    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532845    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532850    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532854    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532858    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532863    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532867    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532871    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532875    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532879    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532883    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532888    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532893    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.548091 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532902    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532906    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532911    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532916    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532920    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532924    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532929    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532933    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532937    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532941    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532945    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532950    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532954    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532963    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532967    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.532972    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533001    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533007    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533012    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533017    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.548780 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533021    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533025    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533029    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533034    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533039    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533043    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533088    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533100    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533108    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533116    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533123    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533128    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533134    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533140    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533146    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.533218    5098 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 03:15:25.549333 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533515    5098 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533530    5098 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533535    5098 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533543    5098 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533550    5098 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533555    5098 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533560    5098 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533564    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533568    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533572    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533576    5098 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533580    5098 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533585    5098 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533589    5098 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533593    5098 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533598    5098 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533602    5098 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533625    5098 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533629    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 03:15:25.549752 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533631    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533634    5098 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533637    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533640    5098 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533644    5098 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533647    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533650    5098 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533653    5098 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533655    5098 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533658    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533661    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533689    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533693    5098 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533696    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533699    5098 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533702    5098 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533705    5098 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533708    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533711    5098 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533713    5098 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 03:15:25.550250 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533716    5098 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533719    5098 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533721    5098 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533724    5098 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533726    5098 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533729    5098 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533732    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533734    5098 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533737    5098 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533740    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533742    5098 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533745    5098 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533748    5098 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533752    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533754    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533757    5098 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533762    5098 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533766    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533769    5098 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533772    5098 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 03:15:25.550761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533775    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533778    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533782    5098 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533785    5098 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533788    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533791    5098 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533794    5098 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533797    5098 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533799    5098 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533802    5098 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533805    5098 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533808    5098 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533810    5098 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533814    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533818    5098 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533822    5098 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533826    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533831    5098 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533835    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533840    5098 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 03:15:25.551299 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533844    5098 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533848    5098 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533853    5098 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533858    5098 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533861    5098 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533879    5098 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533884    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533888    5098 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533892    5098 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533896    5098 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533899    5098 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533902    5098 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533905    5098 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533908    5098 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533910    5098 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533913    5098 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533916    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533921    5098 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533923    5098 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533926    5098 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 03:15:25.551840 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533929    5098 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533932    5098 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533934    5098 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533937    5098 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533940    5098 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533942    5098 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:25.533945    5098 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.534032    5098 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535225    5098 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535309    5098 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535514    5098 server.go:1014] "Starting client certificate rotation"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535653    5098 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535706    5098 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 03:09:43 +0000 UTC" deadline="2028-06-22 07:17:58.906810326 +0000 UTC"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.535783    5098 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14404h2m33.371030404s"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.536405    5098 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.538062    5098 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 03:15:25.552342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.539471    5098 log.go:25] "Validated CRI v1 runtime API"
Oct 31 03:15:25.552840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.541855    5098 log.go:25] "Validated CRI v1 image API"
Oct 31 03:15:25.552840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.542767    5098 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 03:15:25.552840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.545099    5098 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ee241086-c6f9-42c0-8605-e6060086377c:/dev/nvme0n1p4 f8785297-4947-4f7a-bfd3-c17955ae6693:/dev/nvme0n1p3]
Oct 31 03:15:25.552946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.545130    5098 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ff2b7cb7e0a14af04524c4b6f436e7c1c6e7eade3a08ed0c3edda11484d5934/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ff2b7cb7e0a14af04524c4b6f436e7c1c6e7eade3a08ed0c3edda11484d5934/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe/userdata/shm major:0 minor:220 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2/userdata/shm major:0 minor:69 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/801f4d629f6f4621f64db5ec11111ee148d0c08a371eddaba45859b30064e7b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/801f4d629f6f4621f64db5ec11111ee148d0c08a371eddaba45859b30064e7b9/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5/userdata/shm major:0 minor:332 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0736e363-b4be-4a6f-8fe5-e9a1aeea0222/volumes/kubernetes.io~projected/kube-api-access-d7x7d:{mountpoint:/var/lib/kubelet/pods/0736e363-b4be-4a6f-8fe5-e9a1aeea0222/volumes/kubernetes.io~projected/kube-api-access-d7x7d major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/kube-api-access-dfn8t:{mountpoint:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/kube-api-access-dfn8t major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6/volumes/kubernetes.io~projected/kube-api-access-hcz65:{mountpoint:/var/lib/kubelet/pods/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6/volumes/kubernetes.io~projected/kube-api-access-hcz65 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/13947923-5437-4f19-b97d-c6e913cf8fc7/volumes/kubernetes.io~projected/kube-api-access-j5lzv:{mountpoint:/var/lib/kubelet/pods/13947923-5437-4f19-b97d-c6e913cf8fc7/volumes/kubernetes.io~projected/kube-api-access-j5lzv major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~projected/kube-api-access-xv8fs:{mountpoint:/var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~projected/kube-api-access-xv8fs major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~secret/serving-cert major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/231e395b-1984-48a0-865b-b9b05c489eec/volumes/kubernetes.io~projected/kube-api-access-kl7ss:{mountpoint:/var/lib/kubelet/pods/231e395b-1984-48a0-865b-b9b05c489eec/volumes/kubernetes.io~projected/kube-api-access-kl7ss major:0 minor:219 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/290c845c-1024-4d81-a60f-7feb2a3d975a/volumes/kubernetes.io~projected/kube-api-access-spvqm:{mountpoint:/var/lib/kubelet/pods/290c845c-1024-4d81-a60f-7feb2a3d975a/volumes/kubernetes.io~projected/kube-api-access-spvqm major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/29ff3f9f-68da-43f9-a824-b759e77376e3/volumes/kubernetes.io~projected/kube-api-access-kf9lq:{mountpoint:/var/lib/kubelet/pods/29ff3f9f-68da-43f9-a824-b759e77376e3/volumes/kubernetes.io~projected/kube-api-access-kf9lq major:0 minor:329 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/378f839e-767e-421d-b8c7-65369e94a937/volumes/kubernetes.io~projected/kube-api-access-rp8wd:{mountpoint:/var/lib/kubelet/pods/378f839e-767e-421d-b8c7-65369e94a937/volumes/kubernetes.io~projected/kube-api-access-rp8wd major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40423a9f-278f-4f36-87b1-2d05afa56fa7/volumes/kubernetes.io~projected/kube-api-access-bxzfg:{mountpoint:/var/lib/kubelet/pods/40423a9f-278f-4f36-87b1-2d05afa56fa7/volumes/kubernetes.io~projected/kube-api-access-bxzfg major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~projected/kube-api-access-562d5:{mountpoint:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~projected/kube-api-access-562d5 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d7fc633-6f9f-4d5a-adfc-394293b6e4af/volumes/kubernetes.io~projected/kube-api-access-z6lc8:{mountpoint:/var/lib/kubelet/pods/4d7fc633-6f9f-4d5a-adfc-394293b6e4af/volumes/kubernetes.io~projected/kube-api-access-z6lc8 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d97b9da-e0f2-418d-bec3-7ebb1a897a47/volumes/kubernetes.io~projected/kube-api-access-226x7:{mountpoint:/var/lib/kubelet/pods/4d97b9da-e0f2-418d-bec3-7ebb1a897a47/volumes/kubernetes.io~projected/kube-api-access-226x7 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4e4684e1-6f21-46ca-8dec-b41f06386b69/volumes/kubernetes.io~projected/kube-api-access-bgvz8:{mountpoint:/var/lib/kubelet/pods/4e4684e1-6f21-46ca-8dec-b41f06386b69/volumes/kubernetes.io~projected/kube-api-access-bgvz8 major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~projected/kube-api-access-nnsfq:{mountpoint:/var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~projected/kube-api-access-nnsfq major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~secret/serving-cert major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d23fdfb-3765-488e-bad2-d80a7fe3a162/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6d23fdfb-3765-488e-bad2-d80a7fe3a162/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990/volumes/kubernetes.io~projected/kube-api-access-w5ktq:{mountpoint:/var/lib/kubelet/pods/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990/volumes/kubernetes.io~projected/kube-api-access-w5ktq major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~projected/kube-api-access-mxk99:{mountpoint:/var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~projected/kube-api-access-mxk99 major:0 minor:349 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~secret/signing-key major:0 minor:348 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85a63323-7c0f-462b-bdf1-f3a532f2d04c/volumes/kubernetes.io~projected/kube-api-access-zlfgs:{mountpoint:/var/lib/kubelet/pods/85a63323-7c0f-462b-bdf1-f3a532f2d04c/volumes/kubernetes.io~projected/kube-api-access-zlfgs major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/kube-api-access-78kpw:{mountpoint:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/kube-api-access-78kpw major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/97e7fbff-1276-4704-9580-c2a43dc17a97/volumes/kubernetes.io~projected/kube-api-access-5km2s:{mountpoint:/var/lib/kubelet/pods/97e7fbff-1276-4704-9580-c2a43dc17a97/volumes/kubernetes.io~projected/kube-api-access-5km2s major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~projected/kube-api-access-7m979:{mountpoint:/var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~projected/kube-api-access-7m979 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~secret/serving-cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b1d993de-0f1b-4b88-a998-344eec8c23ac/volumes/kubernetes.io~projected/kube-api-access-r2dhc:{mountpoint:/var/lib/kubelet/pods/b1d993de-0f1b-4b88-a998-344eec8c23ac/volumes/kubernetes.io~projected/kube-api-access-r2dhc major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b34e0666-5012-40e2-a7b3-7f6e4470545a/volumes/kubernetes.io~projected/kube-api-access-99jcd:{mountpoint:/var/lib/kubelet/pods/b34e0666-5012-40e2-a7b3-7f6e4470545a/volumes/kubernetes.io~projected/kube-api-access-99jcd major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~projected/kube-api-access-gv8w7:{mountpoint:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~projected/kube-api-access-gv8w7 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/stats-auth major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~projected/kube-api-access-mshws:{mountpoint:/var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~projected/kube-api-access-mshws major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c3f060e6-cd83-4284-98ac-d39a6d8a01be/volumes/kubernetes.io~projected/kube-api-access-kth74:{mountpoint:/var/lib/kubelet/pods/c3f060e6-cd83-4284-98ac-d39a6d8a01be/volumes/kubernetes.io~projected/kube-api-access-kth74 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c81c0c55-4184-4b1b-a431-268e608e3b60/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/c81c0c55-4184-4b1b-a431-268e608e3b60/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:270 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c89742f3-947b-45c2-a14b-491054ba970f/volumes/kubernetes.io~projected/kube-api-access-lg72n:{mountpoint:/var/lib/kubelet/pods/c89742f3-947b-45c2-a14b-491054ba970f/volumes/kubernetes.io~projected/kube-api-access-lg72n major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~projected/kube-api-access-8wzxh:{mountpoint:/var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~projected/kube-api-access-8wzxh major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~secret/cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e077c72b-608e-43e2-b55f-db29e830b507/volumes/kubernetes.io~projected/kube-api-access-cf5nw:{mountpoint:/var/lib/kubelet/pods/e077c72b-608e-43e2-b55f-db29e830b507/volumes/kubernetes.io~projected/kube-api-access-cf5nw major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~projected/kube-api-access-pbvnt:{mountpoint:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~projected/kube-api-access-pbvnt major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdaeb24f-988d-4df9-93ea-4f9c41a672d0/volumes/kubernetes.io~projected/kube-api-access-98bth:{mountpoint:/var/lib/kubelet/pods/fdaeb24f-988d-4df9-93ea-4f9c41a672d0/volumes/kubernetes.io~projected/kube-api-access-98bth major:0 minor:287 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c751f3b637958c6c81ca3482ff070133982db90b68d3f3d84073891f1b3e40ea/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/eeed9d531fe75da7260215a8638d53a6f1e11aa7a1ce5e679ed1c0d720bcde7b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/33b03669f48027f827e44c6311b92b2e4bfe7fa9a333e3fe6905b6d26058edd1/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0237be09a165c335ecd3ce2022efa56e7983ae5d8a09af632711c4b259f41d83/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/72cbe0a4abee4577ba6939ace86612fe2845af92a45b8f598420ee03a1814d67/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/c408648709f420365d26ae393d0e96ef5e82cb115b763267f31b20117ab58729/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9a24851e906fe6dbd49a33add6e9cda29138a77f6d0b32fe254b4cacc6834087/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4053ef466aa2476649f032d036a55dffbcfe54c58356b98ab56ec734167f1870/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/460664ac1e71692a600335cc09e8bf142587d475884552097b32233a5c77e0ba/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5c4daa2fe27ad30d63897677f8f759e77d61e8b13154cf76e0a0972a56557a68/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/9a61597ae083eb89789c437626d73b83ce0655f90498073171bc6bc0adc2d67a/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/770aa90c57c87746c9b2e781a6b406c24d2e2cfde8d2b25664859aa3a93f2ee2/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/3cea0aa8e873f5b9f1fb8e449fd805eb7d26a48e06d76413e0548077c7c27dc5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/5f616537aa02c5a565bfc6ab438ecb9cce55e1181c11aceb9dea8cf3303b3481/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d5e1bfea0aa86eb717e6c352382ea3a6cf120ec919710b6e1948d535d4c37174/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/5aa2bea23b1a0dc89a0a645436a066b6b9e6c21644c2d907e8671214e878fa7d/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/dab548ce693912cc01fc02a9e1eb53a6aba6c50b54a13ca0f5edeb2ecb9c124a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/69a79e081da30fc3575df066e4411fedece7042247d092bc4a88d2972dfa3bd2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/a2cb6bb67dcbab8aabccfb6d7f4f38cbb50218998823766d3352ecf3ec753c99/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0b155311db136dc156678e1eac672e8ac6bb8fa202897f5901a251f44add0c0c/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f0f8a71c2d46db5d293ddeac7376c861dfcd1d68b3f717f5b358dde475cf1607/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/6e7a2fbe066677bfa57e29dad9e628666363f9b406ddb6215e5409261f5a38c2/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/96f98390195cb2e68b758d2253712fd2feac5eca6986ac845e45b87cc384ef29/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/36c8ec85843ce4f2e4cc6374ec2c60883bf7d81a491342eb0f9745867cb260fe/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/bc38afcf545106ebc6d702ccfa368bdbd31b09e6f4f29fcf2f728798dd0f2665/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/c8c73fa7038a10d3ee52d7e1f234f37d9b9a0adf37805cca0655fa2a4876b6e1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/c4cabcc857bb5c505aeb7d9b658932d2dada3cd72935e02c1f5761a36731ffaa/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/1628462564e290ffaeb44e667dd6ac3c1fefecfbb804a47918d3e4ef76fbdea5/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/d192e44ebc283e9de6faa82f2e89ebfe87c3be42020a335641e1db3928dfcb37/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/c29194a67aaa39399048aed6612dbbfaf421fcbfec465c31957cbd5217bf3992/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/7a0691aab00255cb539d8c00eb1109221e4d005c03d1c417b73f65c61c30f595/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/bcf3aeb0655b91dc0194e1e1f896663ec7481903748b5c450e3b5754d85cc011/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/f94c03094bc09084da62b2391ade541eacf8d78431b2a4fd9abb434ccf1ef256/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/b120829047b5e45eb1b2f782f2d987aa1019adf4fa85a7edb67e0856053ad3d8/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/09e2984ad2a9244c6251ca3d375f1808da5b80e866401ab90e5e4486af649a19/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/02b7d24a984f18ce863401f04f4e2f91db57b08f5742ca8e1efd8ea30d2660e3/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/c5fa19e8369ae20957d8457c23fd17805b48813b39f8477574d028cbdbaf4ef1/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/1f7ba0169718f2c36b4a28d4998e2654a43cf35f5587e47125865595db1a2218/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/26f1ba08892b2fc1b580244c7c03cea71f4115b8543cfeb4550d54f0083c6edb/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/3cbc75836469c1b3ea6efc1ff3161cd3533bf35f9c9607c2872aa20edaf87745/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/a53c881561edb127958894f300a1312f97a3f3fb796f477a0efb33db5987756e/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/8fc3055158a0b35ea574ebc231c3adc861172a53bf0740f2364542c1f00aff6e/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/a33a1f6c2245b3fdf17f8273ceac95c546a00d3763888bd780f6a369caf71a46/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/31c25d6bebff4296e9c6f8bad79f9c59723ccca2fa1d65641abd84d64a34f94e/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/f453ca01f7294bb63b3cecd81ae1a4548ad6da308d9b4192d99f6b4c508b5933/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/b591dd726ac01a87e4e503841fcb5707c58f7ae6667dd01ce586b1b91af4a5b3/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/57123e890b93aeed48846eb7d02ea5450a7954ad81746bbebdb980ef2c025de0/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-360:{mountpoint:/var/lib/containers/storage/overlay/124655b7ddea40cbe117128c2274b80068bd832f0f4d2fcbb423a295516f078b/merged major:0 minor:360 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/997cb8e479b529f19c355838ce4c89ea2c1de6adb894845d009d689735c6dee3/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/4a8f6c403978b4386af7b973f438193ecf896c261aa7c6d52767218447a440e6/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/b8509234a0bc4f083aec3c30cb69148e2c8df9f525bf22cf5fc9b9844a121b78/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/8421a58f9618df4a2a8037aba73fdd09985f7e8fad310f752792c1739b293fcb/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-65:{mountpoint:/var/lib/containers/storage/overlay/5eb152803636991dca9ba1dae2c81506ac7f33b58ceb9c225e815c1275513f1f/merged major:0 minor:65 fsType:overlay blockSize:0} overlay_0-67:{mountpoint:/var/lib/containers/storage/overlay/e719c161a8f6eff6b98dde23bbb5a3f27b50bef5de80e7dbb6c1d3f5309324f0/merged major:0 minor:67 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/e8fd45ed05601787008c1e8e2c4a78e505852c01d4d4021d6f94c1d518a7fc8b/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/c76e7e7a661ba829df6b96f606220a5b73c408dd27519ab937f8cf6d4372684b/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/cbe6b3040affaffd3bd877ec336b77411de3c703bafa6a69becde640841ad7e9/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/184d5a071a824069571e8f1b9d31088031d90ede3aff614546be36ea791e23c1/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/7562773d142d9c7d374eb97db16144decfe7ad12cd5c9fb9f1ab084f99e1958f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/8eafd56d173d5f2f83aff0efb21f9359f881adc5d3c87f7cf4e0e98de167eca8/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/a969eb594136c7d2cf22bd9cc2f54ada9edd6c4b48a31636150f30493097914d/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0b483a052742332d9451c3704aadf23b768459d2ec552d443e86d8cbb8fd6c99/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 03:15:25.555501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.554892    5098 manager.go:217] Machine: {Timestamp:2026-10-31 03:15:25.554003883 +0000 UTC m=+0.079223464 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099526 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2731b5aae8aeb67acf3631b2fb3764 SystemUUID:ec2731b5-aae8-aeb6-7acf-3631b2fb3764 BootID:255bf544-b053-406c-befd-6ee08e6ecca2 Filesystems:[{Device:/run/containers/storage/overlay-containers/381737a46175d5ca263413b883aa87b4e0c24cd830fea0d96be59786c82e1b4a/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:208 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~projected/kube-api-access-nnsfq DeviceMajor:0 DeviceMinor:213 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c81c0c55-4184-4b1b-a431-268e608e3b60/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:270 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b1d993de-0f1b-4b88-a998-344eec8c23ac/volumes/kubernetes.io~projected/kube-api-access-r2dhc DeviceMajor:0 DeviceMinor:54 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/231e395b-1984-48a0-865b-b9b05c489eec/volumes/kubernetes.io~projected/kube-api-access-kl7ss DeviceMajor:0 DeviceMinor:219 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d7fc633-6f9f-4d5a-adfc-394293b6e4af/volumes/kubernetes.io~projected/kube-api-access-z6lc8 DeviceMajor:0 DeviceMinor:230 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2156a7ce348c79ba91392618e4177696e7440273b8785537f44bd00501c4a220/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~projected/kube-api-access-mshws DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/97e7fbff-1276-4704-9580-c2a43dc17a97/volumes/kubernetes.io~projected/kube-api-access-5km2s DeviceMajor:0 DeviceMinor:214 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61304d497748223991eb49aa48f96235a47f272738618be1874190788794c0df/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e077c72b-608e-43e2-b55f-db29e830b507/volumes/kubernetes.io~projected/kube-api-access-cf5nw DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c3f060e6-cd83-4284-98ac-d39a6d8a01be/volumes/kubernetes.io~projected/kube-api-access-kth74 DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0736e363-b4be-4a6f-8fe5-e9a1aeea0222/volumes/kubernetes.io~projected/kube-api-access-d7x7d DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:228 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d97b9da-e0f2-418d-bec3-7ebb1a897a47/volumes/kubernetes.io~projected/kube-api-access-226x7 DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dbf0d74024791434d63f68d937748b41e2b3414b9480a881a4684b1a38f9ec19/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c4b754cf4eb1a599d5fd541d8b852a7a78afa7494289a6e02791638b5466d14/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:48 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc41b4e8-af8b-420c-960d-8f948efbf80a/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~projected/kube-api-access-8wzxh DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/6d23fdfb-3765-488e-bad2-d80a7fe3a162/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/68292f3e5ddf36bce36c003c55ab71e79b68c4636a268fb7f4fcf95cc8f97084/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4e4684e1-6f21-46ca-8dec-b41f06386b69/volumes/kubernetes.io~projected/kube-api-access-bgvz8 DeviceMajor:0 DeviceMinor:313 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~projected/kube-api-access-mxk99 DeviceMajor:0 DeviceMinor:349 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/59090ff61eb86592e40dd9e29ed21a6624390045643e86721eaeae92a94010a2/userdata/shm DeviceMajor:0 DeviceMinor:69 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~projected/kube-api-access-7m979 DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/448d389dea2c803eeb5a075a0c5f425e43334351bd2346faac705b15ddf3d271/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/378f839e-767e-421d-b8c7-65369e94a937/volumes/kubernetes.io~projected/kube-api-access-rp8wd DeviceMajor:0 DeviceMinor:218 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ff2b7cb7e0a14af04524c4b6f436e7c1c6e7eade3a08ed0c3edda11484d5934/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6/volumes/kubernetes.io~projected/kube-api-access-hcz65 DeviceMajor:0 DeviceMinor:51 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c0e2d43f9958b0c51c3e1cd754997b9174995686be1d7a33d72283e4c767626/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b34e0666-5012-40e2-a7b3-7f6e4470545a/volumes/kubernetes.io~projected/kube-api-access-99jcd DeviceMajor:0 DeviceMinor:305 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-67 DeviceMajor:0 DeviceMinor:67 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/606fd65b679e05d936572912f059762b699c55d2b90b4cd37438b2c52ddf35af/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6648b0753422529305298cd6cd89006eb3b8cd5f57e640a29e550ff681b46086/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:212 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~projected/kube-api-access-gv8w7 DeviceMajor:0 DeviceMinor:215 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~projected/kube-api-access-78kpw DeviceMajor:0 DeviceMinor:216 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8963bc13ac9a5e5dbcc80b5573bc46767fc3279071ec8b05eb80aa315a5b724e/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14445338d4bffbb4427a3a6a583c8dd9808687eb3eb11113790d0e3c17abe9be/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/231e85e0a7762a2345d10b5072cc162ed69762512b8cc15fe946363eebf5a039/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a585c870e1d7843a58a2f8ad4a33cd6b5de7b9a1f236ce0e75f74ee36b264e5/userdata/shm DeviceMajor:0 DeviceMinor:332 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0312cd4f432cfecc67f33b16584fd5588de3085805f388bd6f48129f52f5749/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:207 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-360 DeviceMajor:0 DeviceMinor:360 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/13947923-5437-4f19-b97d-c6e913cf8fc7/volumes/kubernetes.io~projected/kube-api-access-j5lzv DeviceMajor:0 DeviceMinor:50 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990/volumes/kubernetes.io~projected/kube-api-access-w5ktq DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85a63323-7c0f-462b-bdf1-f3a532f2d04c/volumes/kubernetes.io~projected/kube-api-access-zlfgs DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa4f9f2d-e0c9-46f2-8100-06f434f66322/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4c5cfefbf61b0e94ac1c15e98da3a3f80699438800c2446238ebfdba7d0e22fe/userdata/shm DeviceMajor:0 DeviceMinor:220 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bd351d5cbb21e6188ce22ba49f6391d4f40884ab6739a8c7cde98a349904336/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/859d15d2-a1b9-4c23-b801-8eceb5939a3e/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:348 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~projected/kube-api-access-pbvnt DeviceMajor:0 DeviceMinor:53 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6a509f2cb647d61f7e48a6bd2ec9250785814ab2e8995e78f5e1dee513b3027/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~projected/kube-api-access-dfn8t DeviceMajor:0 DeviceMinor:231 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdaeb24f-988d-4df9-93ea-4f9c41a672d0/volumes/kubernetes.io~projected/kube-api-access-98bth DeviceMajor:0 DeviceMinor:287 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/290c845c-1024-4d81-a60f-7feb2a3d975a/volumes/kubernetes.io~projected/kube-api-access-spvqm DeviceMajor:0 DeviceMinor:330 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/40423a9f-278f-4f36-87b1-2d05afa56fa7/volumes/kubernetes.io~projected/kube-api-access-bxzfg DeviceMajor:0 DeviceMinor:52 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/478f1160-d546-4149-aee6-7a3f6f3f520f/volumes/kubernetes.io~projected/kube-api-access-562d5 DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2029edc37cd9e82bf69e5aba30cfc00e02bf150c6518e6eb8862bdf0f69ce475/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b91cefe7-2e73-4f49-8d39-1187617dc044/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/61a64191-2d4c-48e4-bf08-e010b704cda5/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:209 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3df69533175768ae8a2c5b9a8931c9234307e21f0191caa33ea05152b1f4bfa3/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-65 DeviceMajor:0 DeviceMinor:65 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:227 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:47 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1b4b412a-ef91-4c3e-89c3-9287445d404c/volumes/kubernetes.io~projected/kube-api-access-xv8fs DeviceMajor:0 DeviceMinor:217 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:229 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/29ff3f9f-68da-43f9-a824-b759e77376e3/volumes/kubernetes.io~projected/kube-api-access-kf9lq DeviceMajor:0 DeviceMinor:329 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c89742f3-947b-45c2-a14b-491054ba970f/volumes/kubernetes.io~projected/kube-api-access-lg72n DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/801f4d629f6f4621f64db5ec11111ee148d0c08a371eddaba45859b30064e7b9/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:226 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14445338d4bffbb MacAddress:82:88:8f:66:c6:62 Speed:10000 Mtu:8901} {Name:1bd351d5cbb21e6 MacAddress:b2:c0:c6:6c:01:7c Speed:10000 Mtu:8901} {Name:2156a7ce348c79b MacAddress:5e:dc:12:4d:a4:a0 Speed:10000 Mtu:8901} {Name:231e85e0a7762a2 MacAddress:2a:6b:bb:41:08:2f Speed:10000 Mtu:8901} {Name:3df69533175768a MacAddress:ce:08:f4:4e:53:65 Speed:10000 Mtu:8901} {Name:3ff2b7cb7e0a14a MacAddress:3e:35:fc:0b:59:95 Speed:10000 Mtu:8901} {Name:4c5cfefbf61b0e9 MacAddress:2a:5b:c8:18:13:d7 Speed:10000 Mtu:8901} {Name:606fd65b679e05d MacAddress:22:27:d1:13:9d:92 Speed:10000 Mtu:8901} {Name:61304d497748223 MacAddress:fa:44:87:6a:a7:83 Speed:10000 Mtu:8901} {Name:6648b0753422529 MacAddress:2e:39:c1:7f:e7:71 Speed:10000 Mtu:8901} {Name:6c0e2d43f9958b0 MacAddress:1a:d7:7c:a7:bd:31 Speed:10000 Mtu:8901} {Name:801f4d629f6f462 MacAddress:6e:56:c2:df:5b:35 Speed:10000 Mtu:8901} {Name:8963bc13ac9a5e5 MacAddress:be:19:8f:22:9f:57 Speed:10000 Mtu:8901} {Name:8c4b754cf4eb1a5 MacAddress:36:d7:39:8e:bc:ab Speed:10000 Mtu:8901} {Name:9a585c870e1d784 MacAddress:ce:34:3b:29:c1:4f Speed:10000 Mtu:8901} {Name:ad82f53d7801d1f MacAddress:36:9b:7c:6a:a7:4a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:df:a9:bc:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:fb:29:1e:dc:60 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:df:a9:bc:b7 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:56:ce:cc:5f:b0:19 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:86:fe:4f:bc:04:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 03:15:25.555501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.555508    5098 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 03:15:25.555857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.555573    5098 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 03:15:25.555935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.555899    5098 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 03:15:25.556080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.555937    5098 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-132.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 03:15:25.556127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556083    5098 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 03:15:25.556127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556091    5098 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 03:15:25.556127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556101    5098 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:15:25.556127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556121    5098 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 03:15:25.556228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556141    5098 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 03:15:25.556320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556310    5098 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 03:15:25.556424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556416    5098 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 03:15:25.556461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556452    5098 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 03:15:25.556490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556467    5098 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 03:15:25.556490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556478    5098 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 03:15:25.556490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556487    5098 kubelet.go:405] "Adding apiserver pod source"
Oct 31 03:15:25.556625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.556495    5098 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 03:15:25.557364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.557344    5098 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:15:25.557364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.557361    5098 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:15:25.557967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.557949    5098 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 03:15:25.560191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560174    5098 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 03:15:25.560191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560188    5098 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 03:15:25.560297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560289    5098 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 03:15:25.560489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560478    5098 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 03:15:25.560531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560507    5098 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 03:15:25.560621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560598    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560627    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560633    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560639    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560645    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560651    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560656    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 03:15:25.560659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560662    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560669    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560675    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560684    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560693    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560720    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 03:15:25.560921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.560726    5098 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 03:15:25.564282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.564267    5098 server.go:1263] "Started kubelet"
Oct 31 03:15:25.565012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.564314    5098 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 03:15:25.565012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.564824    5098 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 03:15:25.565012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.564913    5098 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 03:15:25.564928 ip-10-0-129-132 systemd[1]: Started Kubernetes Kubelet.
Oct 31 03:15:25.566869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.565414    5098 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 03:15:25.566869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.566704    5098 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 03:15:25.572203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572182    5098 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 03:15:25.572319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572213    5098 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 03:15:25.572443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572404    5098 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 03:09:44 +0000 UTC" deadline="2028-05-11 23:49:22.962598049 +0000 UTC"
Oct 31 03:15:25.572505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572443    5098 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13412h33m57.390158276s"
Oct 31 03:15:25.572505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572476    5098 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 03:15:25.572505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572483    5098 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 03:15:25.572699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.572630    5098 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 03:15:25.573439 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.573356    5098 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-132.us-east-2.compute.internal\" not found"
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.574188    5098 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575143    5098 factory.go:55] Registering systemd factory
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575164    5098 factory.go:223] Registration of the systemd container factory successfully
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575384    5098 factory.go:153] Registering CRI-O factory
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575397    5098 factory.go:223] Registration of the crio container factory successfully
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575445    5098 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575465    5098 factory.go:103] Registering Raw factory
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575479    5098 manager.go:1199] Started watching for new ooms in manager
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.575845    5098 manager.go:319] Starting recovery of all containers
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576317    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="231e395b-1984-48a0-865b-b9b05c489eec" volumeName="kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576345    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378f839e-767e-421d-b8c7-65369e94a937" volumeName="kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576356    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859d15d2-a1b9-4c23-b801-8eceb5939a3e" volumeName="kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576364    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576373    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576381    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576388    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d993de-0f1b-4b88-a998-344eec8c23ac" volumeName="kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576396    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576405    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576412    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4f9f2d-e0c9-46f2-8100-06f434f66322" volumeName="kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576421    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576428    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3f060e6-cd83-4284-98ac-d39a6d8a01be" volumeName="kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576435    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c89742f3-947b-45c2-a14b-491054ba970f" volumeName="kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576443    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de7d30b6-5160-4f1b-a67b-a0c92f02e0fa" volumeName="kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576452    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="de7d30b6-5160-4f1b-a67b-a0c92f02e0fa" volumeName="kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576459    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4b412a-ef91-4c3e-89c3-9287445d404c" volumeName="kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576467    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4b412a-ef91-4c3e-89c3-9287445d404c" volumeName="kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576477    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576485    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576507    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d23fdfb-3765-488e-bad2-d80a7fe3a162" volumeName="kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576515    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79a249fe-49bc-4f12-b1f5-c2e4b3ecc990" volumeName="kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576523    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859d15d2-a1b9-4c23-b801-8eceb5939a3e" volumeName="kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576531    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34e0666-5012-40e2-a7b3-7f6e4470545a" volumeName="kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576540    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91cefe7-2e73-4f49-8d39-1187617dc044" volumeName="kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576547    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092c4d2f-ae5c-424d-b458-d19bf1e0e8f6" volumeName="kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576569    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7fc633-6f9f-4d5a-adfc-394293b6e4af" volumeName="kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576587    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34e0666-5012-40e2-a7b3-7f6e4470545a" volumeName="kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576595    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576624    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576636    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d97b9da-e0f2-418d-bec3-7ebb1a897a47" volumeName="kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576647    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a64191-2d4c-48e4-bf08-e010b704cda5" volumeName="kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576658    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b2dbb0-3454-4290-81ef-3cf35b21c1d0" volumeName="kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned" seLinuxMountContext=""
Oct 31 03:15:25.576644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576669    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f1160-d546-4149-aee6-7a3f6f3f520f" volumeName="kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576683    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f1160-d546-4149-aee6-7a3f6f3f520f" volumeName="kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576694    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576705    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4b412a-ef91-4c3e-89c3-9287445d404c" volumeName="kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576715    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b1d993de-0f1b-4b88-a998-344eec8c23ac" volumeName="kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576726    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576737    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e077c72b-608e-43e2-b55f-db29e830b507" volumeName="kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576749    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdaeb24f-988d-4df9-93ea-4f9c41a672d0" volumeName="kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576762    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4684e1-6f21-46ca-8dec-b41f06386b69" volumeName="kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576773    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4f9f2d-e0c9-46f2-8100-06f434f66322" volumeName="kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576784    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34e0666-5012-40e2-a7b3-7f6e4470545a" volumeName="kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576795    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576806    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f1160-d546-4149-aee6-7a3f6f3f520f" volumeName="kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576821    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f1160-d546-4149-aee6-7a3f6f3f520f" volumeName="kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576835    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="478f1160-d546-4149-aee6-7a3f6f3f520f" volumeName="kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576848    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7fc633-6f9f-4d5a-adfc-394293b6e4af" volumeName="kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576860    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa4f9f2d-e0c9-46f2-8100-06f434f66322" volumeName="kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576874    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576887    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="378f839e-767e-421d-b8c7-65369e94a937" volumeName="kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576898    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d7fc633-6f9f-4d5a-adfc-394293b6e4af" volumeName="kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576914    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092c4d2f-ae5c-424d-b458-d19bf1e0e8f6" volumeName="kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576926    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="13947923-5437-4f19-b97d-c6e913cf8fc7" volumeName="kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576938    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85a63323-7c0f-462b-bdf1-f3a532f2d04c" volumeName="kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576950    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85a63323-7c0f-462b-bdf1-f3a532f2d04c" volumeName="kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576971    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576985    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0736e363-b4be-4a6f-8fe5-e9a1aeea0222" volumeName="kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.576997    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577009    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="074d5926-982b-4fcc-913c-682b38ab3683" volumeName="kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577060    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a64191-2d4c-48e4-bf08-e010b704cda5" volumeName="kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577074    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d23fdfb-3765-488e-bad2-d80a7fe3a162" volumeName="kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577087    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85a63323-7c0f-462b-bdf1-f3a532f2d04c" volumeName="kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577099    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577113    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b2dbb0-3454-4290-81ef-3cf35b21c1d0" volumeName="kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577126    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4e4684e1-6f21-46ca-8dec-b41f06386b69" volumeName="kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577139    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97e7fbff-1276-4704-9580-c2a43dc17a97" volumeName="kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577151    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a4b3f3c-5b83-481b-9038-aad4508357bf" volumeName="kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577164    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c81c0c55-4184-4b1b-a431-268e608e3b60" volumeName="kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577177    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="61a64191-2d4c-48e4-bf08-e010b704cda5" volumeName="kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577191    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85a63323-7c0f-462b-bdf1-f3a532f2d04c" volumeName="kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577202    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="97322fdd-f911-47eb-b1b5-c336eb539fbe" volumeName="kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577214    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c3f060e6-cd83-4284-98ac-d39a6d8a01be" volumeName="kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577226    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="290c845c-1024-4d81-a60f-7feb2a3d975a" volumeName="kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577238    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40423a9f-278f-4f36-87b1-2d05afa56fa7" volumeName="kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577249    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="859d15d2-a1b9-4c23-b801-8eceb5939a3e" volumeName="kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577262    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b34e0666-5012-40e2-a7b3-7f6e4470545a" volumeName="kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577275    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc41b4e8-af8b-420c-960d-8f948efbf80a" volumeName="kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577288    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3b2dbb0-3454-4290-81ef-3cf35b21c1d0" volumeName="kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577300    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="092c4d2f-ae5c-424d-b458-d19bf1e0e8f6" volumeName="kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577313    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1b4b412a-ef91-4c3e-89c3-9287445d404c" volumeName="kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577325    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="29ff3f9f-68da-43f9-a824-b759e77376e3" volumeName="kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577339    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79a249fe-49bc-4f12-b1f5-c2e4b3ecc990" volumeName="kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577351    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91cefe7-2e73-4f49-8d39-1187617dc044" volumeName="kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577363    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b91cefe7-2e73-4f49-8d39-1187617dc044" volumeName="kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577376    5098 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="231e395b-1984-48a0-865b-b9b05c489eec" volumeName="kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config" seLinuxMountContext=""
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577388    5098 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 03:15:25.578671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.577395    5098 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 03:15:25.581208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.578971    5098 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:15:25.581208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.579091    5098 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:15:25.582270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.582239    5098 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 03:15:25.592806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.592778    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:25.593535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.593510    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:25.601369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.601324    5098 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 03:15:25.603044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.603018    5098 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 03:15:25.603044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.603046    5098 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 03:15:25.603246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.603073    5098 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 03:15:25.603246 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.603124    5098 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 03:15:25.607750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.607719    5098 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 03:15:25.615457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.615427    5098 generic.go:356] "Generic (PLEG): container finished" podID="dcbfbb892bae597c340f8a0c3dc3499e" containerID="2b35d65b1a4cdbae5039a6bc5a405516101081d294e8fa3140c4d48e448eede0" exitCode=0
Oct 31 03:15:25.635877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.635852    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:25.636032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.635885    5098 generic.go:356] "Generic (PLEG): container finished" podID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421" exitCode=255
Oct 31 03:15:25.659442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659407    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="29927f1f75f5a7aa60440ae0029e4f214727c5249263effabf7070df9e829d2b" exitCode=0
Oct 31 03:15:25.659442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659435    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="d1c926738ce57fe16342192d05406a9c1038943ad3ef518164c96b78c75c99ca" exitCode=0
Oct 31 03:15:25.659442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659442    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="99de9cc71e1bf533620491cc178b7a8a940df0fcce7740bc49198751d2d6ab47" exitCode=0
Oct 31 03:15:25.659442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659450    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="48bf176727a45de28bdc3e8519d34fca13be87855d435db5f914062c7319f7da" exitCode=0
Oct 31 03:15:25.659442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659455    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="8afcd5608e097fec33be404d3742a31800fd30299457fe84356a565f8ab3f36e" exitCode=0
Oct 31 03:15:25.659809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.659462    5098 generic.go:356] "Generic (PLEG): container finished" podID="85a63323-7c0f-462b-bdf1-f3a532f2d04c" containerID="1be4dc60c0c2987477f4a9a94083e3df3c1fa541e12ccc6ae9a8240e24b92ac0" exitCode=0
Oct 31 03:15:25.678435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.678364    5098 manager.go:324] Recovery completed
Oct 31 03:15:25.702395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702365    5098 cpu_manager.go:226] "Starting" policy="none"
Oct 31 03:15:25.702395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702385    5098 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702409    5098 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702539    5098 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702549    5098 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702569    5098 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702575    5098 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702580    5098 policy_none.go:50] "Start"
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702588    5098 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 03:15:25.702646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702598    5098 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 03:15:25.702880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702769    5098 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 03:15:25.702880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702778    5098 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 03:15:25.702880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.702785    5098 policy_none.go:44] "Start"
Oct 31 03:15:25.703996 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.703975    5098 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Oct 31 03:15:25.707507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.707452    5098 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 03:15:25.711412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.711394    5098 manager.go:341] "Starting Device Plugin manager"
Oct 31 03:15:25.712075 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.711429    5098 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 03:15:25.712075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712064    5098 server.go:87] "Starting device plugin registration server"
Oct 31 03:15:25.712342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712326    5098 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 03:15:25.712420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712343    5098 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 03:15:25.712505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712487    5098 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 03:15:25.712656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712592    5098 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 03:15:25.712729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712661    5098 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 03:15:25.712729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712667    5098 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 03:15:25.712850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.712756    5098 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T03:15:25.712650287Z","UUID":"05f41b4a-11cf-4df1-b7c0-6a72035898a7","Handler":null,"Name":"","Endpoint":""}
Oct 31 03:15:25.714161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.714143    5098 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 03:15:25.714266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.714180    5098 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 03:15:25.714266 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.714242    5098 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 03:15:25.814529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.814478    5098 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 03:15:25.815985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.815967    5098 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientMemory"
Oct 31 03:15:25.816082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.815998    5098 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 03:15:25.816082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.816010    5098 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-132.us-east-2.compute.internal" event="NodeHasSufficientPID"
Oct 31 03:15:25.816082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.816055    5098 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.829080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.829055    5098 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.829212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.829127    5098 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.905066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.904983    5098 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"]
Oct 31 03:15:25.905776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905731    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerStarted","Data":"b49d20fc9ddacadc24b7e0c2aeb106b1ee0bc9da529d873b0112d46326c1182b"}
Oct 31 03:15:25.905891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905776    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerDied","Data":"2b35d65b1a4cdbae5039a6bc5a405516101081d294e8fa3140c4d48e448eede0"}
Oct 31 03:15:25.905891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905802    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"dcbfbb892bae597c340f8a0c3dc3499e","Type":"ContainerStarted","Data":"f657b0cb49028e376414c366c96bb9aabbaca4d8524e69aaf05b9875fe631c61"}
Oct 31 03:15:25.905891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905819    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"776064ad5d3782b191602f1d077b5a65","Type":"ContainerStarted","Data":"1c8744fa95dc6ce578ecd40b6e873d41992aa8ea660339c4cd53dd5b972d3b15"}
Oct 31 03:15:25.905891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905835    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal" event={"ID":"776064ad5d3782b191602f1d077b5a65","Type":"ContainerStarted","Data":"27eaa786d6fbae782014333e978c7effba23b9559a9ba6a2cdb0f3e9ce88e6ba"}
Oct 31 03:15:25.906109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905934    5098 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.906109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.905981    5098 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.919716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.919687    5098 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:15:25.919850 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.919754    5098 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.920112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.920094    5098 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:15:25.920170 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:25.920135    5098 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.984562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.984480    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.984562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.984521    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:25.984763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:25.984569    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085100    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085149    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085177    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085218    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085253    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dcbfbb892bae597c340f8a0c3dc3499e-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"dcbfbb892bae597c340f8a0c3dc3499e\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.085317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.085225    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/776064ad5d3782b191602f1d077b5a65-config\") pod \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" (UID: \"776064ad5d3782b191602f1d077b5a65\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.557004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.556968    5098 apiserver.go:52] "Watching apiserver"
Oct 31 03:15:26.567799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.567772    5098 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 03:15:26.568578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.568532    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4crls","openshift-image-registry/node-ca-wtqp4","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal","openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n","openshift-image-registry/image-registry-859c7cc5c7-8q95x","openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4","kube-system/kas-connection-checker-685895c7c9-5f46r","openshift-cluster-node-tuning-operator/tuned-jmgbj","openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b","openshift-multus/network-metrics-daemon-bmxnp","kube-system/konnectivity-agent-6csjw","openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv","openshift-service-ca/service-ca-5fffc97bdf-sqvnc","openshift-insights/periodic-gathering-278hz-d9lkp","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms","openshift-multus/multus-additional-cni-plugins-jjjbw","openshift-network-operator/iptables-alerter-xrkgp","openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f","kube-system/kas-connection-checker-685895c7c9-dkwqg","openshift-network-diagnostics/network-check-target-4ww6v","openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x","kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-5fgwn","kube-system/global-pull-secret-syncer-bjxm9","openshift-network-console/networking-console-plugin-86c478bb88-4927g","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k","openshift-ingress-canary/ingress-canary-ft2z2","openshift-multus/multus-7p689","openshift-console-operator/console-operator-7cd56b8dbc-nq5jp","openshift-ingress/router-default-5497575d9c-lm8wr","kube-system/kas-connection-checker-685895c7c9-jmx67","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7","openshift-dns/node-resolver-nnxgp","openshift-insights/insights-operator-7c4986d449-j2gqr","openshift-dns/dns-default-lbdjh"]
Oct 31 03:15:26.570912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.570887    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.572836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.572809    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.573241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.573216    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.573367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.573267    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.573367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.573343    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.573912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.573889    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.574210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.574189    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.575853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.575828    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Oct 31 03:15:26.575945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.575922    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Oct 31 03:15:26.576217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.576200    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-z4k9w\""
Oct 31 03:15:26.576705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.576678    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Oct 31 03:15:26.576865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.576847    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.577386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577364    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Oct 31 03:15:26.577461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577418    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.577620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577591    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Oct 31 03:15:26.577696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577677    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.577820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577806    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.577993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.577971    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.578299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.578278    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.578946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.578919    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.579043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579019    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 03:15:26.579290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579268    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Oct 31 03:15:26.579353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579295    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hcpdk\""
Oct 31 03:15:26.579415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579352    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.579415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579395    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.579509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579445    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.579553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.578285    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.579690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579668    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gz7pd\""
Oct 31 03:15:26.579764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579700    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 03:15:26.579820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.579773    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 03:15:26.580642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580598    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.580642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580629    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.580778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580722    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 03:15:26.580915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580901    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Oct 31 03:15:26.581004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580951    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.581004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.580977    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Oct 31 03:15:26.581427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581409    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.581427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581411    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Oct 31 03:15:26.581586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581540    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.581730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581715    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Oct 31 03:15:26.581837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581818    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.581909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581897    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.581976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581963    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Oct 31 03:15:26.582023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.581967    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9h5qx\""
Oct 31 03:15:26.582072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582035    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Oct 31 03:15:26.582363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582341    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.582466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582455    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.582581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582563    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-c7fps\""
Oct 31 03:15:26.582581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582571    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.582724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582664    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mv56x\""
Oct 31 03:15:26.582724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582716    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 03:15:26.582807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582753    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\""
Oct 31 03:15:26.582868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.582816    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Oct 31 03:15:26.583184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.583163    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Oct 31 03:15:26.583532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.583512    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 03:15:26.583700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.583682    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Oct 31 03:15:26.583865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.583851    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Oct 31 03:15:26.583941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.583928    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-2x9p7\""
Oct 31 03:15:26.584082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.584068    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 03:15:26.585019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.585004    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Oct 31 03:15:26.586510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586489    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.586675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586599    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 03:15:26.586675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586628    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf5nw\" (UniqueName: \"kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw\") pod \"kas-connection-checker-685895c7c9-dkwqg\" (UID: \"e077c72b-608e-43e2-b55f-db29e830b507\") " pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:26.586675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586644    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.586675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586658    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586734    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-snapshots\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586761    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lg72n\" (UniqueName: \"kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n\") pod \"kas-connection-checker-685895c7c9-jmx67\" (UID: \"c89742f3-947b-45c2-a14b-491054ba970f\") " pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586794    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586829    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586853    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8w7\" (UniqueName: \"kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586870    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586886    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-226x7\" (UniqueName: \"kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7\") pod \"network-check-source-6f78d59f99-29k8x\" (UID: \"4d97b9da-e0f2-418d-bec3-7ebb1a897a47\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:26.586920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586907    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586936    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586964    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586982    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa4f9f2d-e0c9-46f2-8100-06f434f66322-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.586991    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587015    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587039    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8fs\" (UniqueName: \"kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587053    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-stats-auth\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587086    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587133    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m979\" (UniqueName: \"kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587193    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587223    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5km2s\" (UniqueName: \"kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s\") pod \"volume-data-source-validator-58c9f5f46d-ktx5k\" (UID: \"97e7fbff-1276-4704-9580-c2a43dc17a97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587253    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7x7d\" (UniqueName: \"kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d\") pod \"kas-connection-checker-685895c7c9-5f46r\" (UID: \"0736e363-b4be-4a6f-8fe5-e9a1aeea0222\") " pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587290    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-default-certificate\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587290    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa4f9f2d-e0c9-46f2-8100-06f434f66322-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587307    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587322    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-service-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587336    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp8wd\" (UniqueName: \"kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587439    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mshws\" (UniqueName: \"kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587443    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bc41b4e8-af8b-420c-960d-8f948efbf80a-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587442    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bc41b4e8-af8b-420c-960d-8f948efbf80a-serving-cert\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587480    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587307    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/378f839e-767e-421d-b8c7-65369e94a937-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.587534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587518    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587566    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587619    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-trusted-ca\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587655    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587698    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc41b4e8-af8b-420c-960d-8f948efbf80a-tmp\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587772    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587825    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b4b412a-ef91-4c3e-89c3-9287445d404c-serving-cert\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587844    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.587962    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b4b412a-ef91-4c3e-89c3-9287445d404c-config\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.588089    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k8hk8\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.588503    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-r6lm2\""
Oct 31 03:15:26.588917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.588669    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nklpg\""
Oct 31 03:15:26.589675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.589468    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 03:15:26.589866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.589849    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.590103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590080    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 03:15:26.590371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590344    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-b55zh\""
Oct 31 03:15:26.590371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590356    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.590530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590385    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 03:15:26.590754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590737    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 03:15:26.590841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.590766    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 03:15:26.591023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591007    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 03:15:26.591267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591250    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-fkjgm\""
Oct 31 03:15:26.591386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591369    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.591444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591401    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.591593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591369    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.591706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591597    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-qg7r7\""
Oct 31 03:15:26.591827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591806    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 03:15:26.592005 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.591989    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-d5pxc\""
Oct 31 03:15:26.592195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592172    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 03:15:26.592365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592338    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 03:15:26.592525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592500    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 03:15:26.592590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592571    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 03:15:26.592590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592580    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rfxnp\""
Oct 31 03:15:26.592743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592727    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fn5cc\""
Oct 31 03:15:26.592851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.592837    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gmjq8\""
Oct 31 03:15:26.593795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.593021    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.593795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.593113    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.593795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.593134    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5s7vx\""
Oct 31 03:15:26.593795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.593251    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-rtvr4\""
Oct 31 03:15:26.595112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595094    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.595173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595098    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf5nw\" (UniqueName: \"kubernetes.io/projected/e077c72b-608e-43e2-b55f-db29e830b507-kube-api-access-cf5nw\") pod \"kas-connection-checker-685895c7c9-dkwqg\" (UID: \"e077c72b-608e-43e2-b55f-db29e830b507\") " pod="kube-system/kas-connection-checker-685895c7c9-dkwqg"
Oct 31 03:15:26.595217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595175    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.595439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595416    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-226x7\" (UniqueName: \"kubernetes.io/projected/4d97b9da-e0f2-418d-bec3-7ebb1a897a47-kube-api-access-226x7\") pod \"network-check-source-6f78d59f99-29k8x\" (UID: \"4d97b9da-e0f2-418d-bec3-7ebb1a897a47\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-29k8x"
Oct 31 03:15:26.595529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595514    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.595573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.595513    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.596046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.596027    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8w7\" (UniqueName: \"kubernetes.io/projected/b91cefe7-2e73-4f49-8d39-1187617dc044-kube-api-access-gv8w7\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.596122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.596026    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m979\" (UniqueName: \"kubernetes.io/projected/aa4f9f2d-e0c9-46f2-8100-06f434f66322-kube-api-access-7m979\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-w6tms\" (UID: \"aa4f9f2d-e0c9-46f2-8100-06f434f66322\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms"
Oct 31 03:15:26.596562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.596543    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp8wd\" (UniqueName: \"kubernetes.io/projected/378f839e-767e-421d-b8c7-65369e94a937-kube-api-access-rp8wd\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.596665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.596644    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lg72n\" (UniqueName: \"kubernetes.io/projected/c89742f3-947b-45c2-a14b-491054ba970f-kube-api-access-lg72n\") pod \"kas-connection-checker-685895c7c9-jmx67\" (UID: \"c89742f3-947b-45c2-a14b-491054ba970f\") " pod="kube-system/kas-connection-checker-685895c7c9-jmx67"
Oct 31 03:15:26.596985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.596967    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8fs\" (UniqueName: \"kubernetes.io/projected/1b4b412a-ef91-4c3e-89c3-9287445d404c-kube-api-access-xv8fs\") pod \"console-operator-7cd56b8dbc-nq5jp\" (UID: \"1b4b412a-ef91-4c3e-89c3-9287445d404c\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:26.597279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597259    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 03:15:26.597346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597300    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.597346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597306    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.597346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597324    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-zf8rk\""
Oct 31 03:15:26.597552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597520    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 03:15:26.597849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597831    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7x7d\" (UniqueName: \"kubernetes.io/projected/0736e363-b4be-4a6f-8fe5-e9a1aeea0222-kube-api-access-d7x7d\") pod \"kas-connection-checker-685895c7c9-5f46r\" (UID: \"0736e363-b4be-4a6f-8fe5-e9a1aeea0222\") " pod="kube-system/kas-connection-checker-685895c7c9-5f46r"
Oct 31 03:15:26.597972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.597951    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5km2s\" (UniqueName: \"kubernetes.io/projected/97e7fbff-1276-4704-9580-c2a43dc17a97-kube-api-access-5km2s\") pod \"volume-data-source-validator-58c9f5f46d-ktx5k\" (UID: \"97e7fbff-1276-4704-9580-c2a43dc17a97\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-ktx5k"
Oct 31 03:15:26.598181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598160    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 03:15:26.598325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598310    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mshws\" (UniqueName: \"kubernetes.io/projected/bc41b4e8-af8b-420c-960d-8f948efbf80a-kube-api-access-mshws\") pod \"insights-operator-7c4986d449-j2gqr\" (UID: \"bc41b4e8-af8b-420c-960d-8f948efbf80a\") " pod="openshift-insights/insights-operator-7c4986d449-j2gqr"
Oct 31 03:15:26.598415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598402    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-rrfxr\""
Oct 31 03:15:26.598507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598493    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dk5h8\""
Oct 31 03:15:26.598783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598768    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 03:15:26.598860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598797    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dct2q\""
Oct 31 03:15:26.598860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.598799    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 03:15:26.599712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.599700    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.599782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.599768    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-4hz9x\""
Oct 31 03:15:26.599818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.599773    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.601141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.601130    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 03:15:26.601375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.601363    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 03:15:26.601498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.601487    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-q4pxs\""
Oct 31 03:15:26.601562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.601548    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 03:15:26.601600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.601565    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 03:15:26.671107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.671077    5098 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.671271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.671202    5098 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.674751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.674734    5098 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 03:15:26.683745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.683713    5098 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:15:26.683847 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.683771    5098 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.683847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.683797    5098 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 03:15:26.683847 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.683841    5098 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal"
Oct 31 03:15:26.687925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.687909    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.687964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.687936    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.687964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.687955    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5ktq\" (UniqueName: \"kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.688045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.687971    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.688104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688044    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.688104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.688203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688186    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.688245 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688215    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.688298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688274    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9lq\" (UniqueName: \"kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq\") pod \"migrator-66df5d496b-2wr8b\" (UID: \"29ff3f9f-68da-43f9-a824-b759e77376e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:26.688345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688325    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.688410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688391    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.688470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688449    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-config\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.688663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688456    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.688663 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.688516    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.188494361 +0000 UTC m=+1.713713952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:26.688663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688541    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dhc\" (UniqueName: \"kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.688663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688568    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgvz8\" (UniqueName: \"kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.688663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688596    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:26.688912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688746    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.688912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688785    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.688912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688829    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.688912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688847    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.688912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688871    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlfgs\" (UniqueName: \"kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688924    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688931    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6d23fdfb-3765-488e-bad2-d80a7fe3a162-konnectivity-ca\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.688976    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689004    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689021    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.689149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689080    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689165    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689165    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689236    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689260    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689292    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689332    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689373    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689373    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-config-volume\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689394    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-tmp\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689394    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-tuned\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.689434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689418    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689452    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689483    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689517    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689544    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689570    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689731    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689765    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6lc8\" (UniqueName: \"kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689791    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.689818    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690041    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcz65\" (UniqueName: \"kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690074    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690101    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690127    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690127    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690182    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690181    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-daemon-config\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690205    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690342    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.690408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690401    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690436    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690480    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690537    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690594    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690712    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690740    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690765    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690784    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-ovnkube-script-lib\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690799    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690827    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-cabundle\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.690873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690851    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690901    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690931    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.690984    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/478f1160-d546-4149-aee6-7a3f6f3f520f-env-overrides\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691018    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691021    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691076    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691098    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691116    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691119    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691205    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691232    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691244    5098 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691277    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691286    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6d23fdfb-3765-488e-bad2-d80a7fe3a162-agent-certs\") pod \"konnectivity-agent-6csjw\" (UID: \"6d23fdfb-3765-488e-bad2-d80a7fe3a162\") " pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691320    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnsfq\" (UniqueName: \"kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1d993de-0f1b-4b88-a998-344eec8c23ac-tmp-dir\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.691340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691348    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691374    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691406    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691478    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-cert\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691482    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691518    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvnt\" (UniqueName: \"kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691537    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4e4684e1-6f21-46ca-8dec-b41f06386b69-data-volume\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691548    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691852    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691871    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691768    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c3f060e6-cd83-4284-98ac-d39a6d8a01be-iptables-alerter-script\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.691975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.691897    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wzxh\" (UniqueName: \"kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692035    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692088    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692112    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692130    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692146    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692145    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/61a64191-2d4c-48e4-bf08-e010b704cda5-config\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692192    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692280    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692323    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692346    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.692369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692366    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cni-binary-copy\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692371    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692415    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692445    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spvqm\" (UniqueName: \"kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm\") pod \"migrator-66df5d496b-xkkh4\" (UID: \"290c845c-1024-4d81-a60f-7feb2a3d975a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692594    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692651    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/859d15d2-a1b9-4c23-b801-8eceb5939a3e-signing-key\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692662    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692818    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.692841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692837    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cni-binary-copy\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692868    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692911    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692927    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kth74\" (UniqueName: \"kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.692970    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693013    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693071    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693130    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.693159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693126    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693170    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693202    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693248    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693290    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxzfg\" (UniqueName: \"kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693314    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693338    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693363    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.693530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693473    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c81c0c55-4184-4b1b-a431-268e608e3b60-original-pull-secret\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.693785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693387    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.693785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693673    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.693785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693703    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.693785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693730    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.693785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693771    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693787    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693836    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/478f1160-d546-4149-aee6-7a3f6f3f520f-ovn-node-metrics-cert\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693847    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693882    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693914    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.693960    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.694012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694006    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694030    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694086    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694131    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694164    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxk99\" (UniqueName: \"kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694196    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694229    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl7ss\" (UniqueName: \"kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.694296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694269    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694312    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-tmp-dir\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694309    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5lzv\" (UniqueName: \"kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694365    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694392    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694393    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694363    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/85a63323-7c0f-462b-bdf1-f3a532f2d04c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694516    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.694635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694600    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-serviceca\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694620    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694678    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694713    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694765    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694800    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-562d5\" (UniqueName: \"kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694838    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.694975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.694928    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/61a64191-2d4c-48e4-bf08-e010b704cda5-serving-cert\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.695290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.695045    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.695290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.695046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91cefe7-2e73-4f49-8d39-1187617dc044-metrics-certs\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:26.695290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.695166    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.695419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.695337    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a4b3f3c-5b83-481b-9038-aad4508357bf-nginx-conf\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.695591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.695574    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.697085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.697068    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/378f839e-767e-421d-b8c7-65369e94a937-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-2cg4n\" (UID: \"378f839e-767e-421d-b8c7-65369e94a937\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.697395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.697376    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9lq\" (UniqueName: \"kubernetes.io/projected/29ff3f9f-68da-43f9-a824-b759e77376e3-kube-api-access-kf9lq\") pod \"migrator-66df5d496b-2wr8b\" (UID: \"29ff3f9f-68da-43f9-a824-b759e77376e3\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-2wr8b"
Oct 31 03:15:26.697495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.697478    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgvz8\" (UniqueName: \"kubernetes.io/projected/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-api-access-bgvz8\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.698695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.698669    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5ktq\" (UniqueName: \"kubernetes.io/projected/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-kube-api-access-w5ktq\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.698777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.698761    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.700362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700015    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6lc8\" (UniqueName: \"kubernetes.io/projected/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-kube-api-access-z6lc8\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.700362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700218    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dhc\" (UniqueName: \"kubernetes.io/projected/b1d993de-0f1b-4b88-a998-344eec8c23ac-kube-api-access-r2dhc\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.700362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700271    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlfgs\" (UniqueName: \"kubernetes.io/projected/85a63323-7c0f-462b-bdf1-f3a532f2d04c-kube-api-access-zlfgs\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.700362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700329    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnsfq\" (UniqueName: \"kubernetes.io/projected/61a64191-2d4c-48e4-bf08-e010b704cda5-kube-api-access-nnsfq\") pod \"service-ca-operator-544d749c8c-vwf4f\" (UID: \"61a64191-2d4c-48e4-bf08-e010b704cda5\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f"
Oct 31 03:15:26.700565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700547    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcz65\" (UniqueName: \"kubernetes.io/projected/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-kube-api-access-hcz65\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.700688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.700673    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.702135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.702119    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spvqm\" (UniqueName: \"kubernetes.io/projected/290c845c-1024-4d81-a60f-7feb2a3d975a-kube-api-access-spvqm\") pod \"migrator-66df5d496b-xkkh4\" (UID: \"290c845c-1024-4d81-a60f-7feb2a3d975a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xkkh4"
Oct 31 03:15:26.703669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.703027    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl7ss\" (UniqueName: \"kubernetes.io/projected/231e395b-1984-48a0-865b-b9b05c489eec-kube-api-access-kl7ss\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.703834    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"periodic-gathering-278hz-d9lkp\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") " pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.704077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5lzv\" (UniqueName: \"kubernetes.io/projected/13947923-5437-4f19-b97d-c6e913cf8fc7-kube-api-access-j5lzv\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.704343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxk99\" (UniqueName: \"kubernetes.io/projected/859d15d2-a1b9-4c23-b801-8eceb5939a3e-kube-api-access-mxk99\") pod \"service-ca-5fffc97bdf-sqvnc\" (UID: \"859d15d2-a1b9-4c23-b801-8eceb5939a3e\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sqvnc"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.704394    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kth74\" (UniqueName: \"kubernetes.io/projected/c3f060e6-cd83-4284-98ac-d39a6d8a01be-kube-api-access-kth74\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.704676    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.704911    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.705127    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvnt\" (UniqueName: \"kubernetes.io/projected/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-kube-api-access-pbvnt\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.705172    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxzfg\" (UniqueName: \"kubernetes.io/projected/40423a9f-278f-4f36-87b1-2d05afa56fa7-kube-api-access-bxzfg\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.705421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.705259    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wzxh\" (UniqueName: \"kubernetes.io/projected/de7d30b6-5160-4f1b-a67b-a0c92f02e0fa-kube-api-access-8wzxh\") pod \"ingress-canary-ft2z2\" (UID: \"de7d30b6-5160-4f1b-a67b-a0c92f02e0fa\") " pod="openshift-ingress-canary/ingress-canary-ft2z2"
Oct 31 03:15:26.708698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.706437    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98bth\" (UniqueName: \"kubernetes.io/projected/fdaeb24f-988d-4df9-93ea-4f9c41a672d0-kube-api-access-98bth\") pod \"network-check-target-4ww6v\" (UID: \"fdaeb24f-988d-4df9-93ea-4f9c41a672d0\") " pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:26.708698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.706697    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-562d5\" (UniqueName: \"kubernetes.io/projected/478f1160-d546-4149-aee6-7a3f6f3f520f-kube-api-access-562d5\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796370    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:26.796409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796408    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796426    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796443    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796459    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796475    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796492    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796515    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796516    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-systemd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796536    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.796546    5098 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796538    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796564    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796580    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-slash\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796572    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796597    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796619    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-ovn\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796632    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-modprobe-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796634    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-d\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.796644    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.296622124 +0000 UTC m=+1.821841711 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796647    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79a249fe-49bc-4f12-b1f5-c2e4b3ecc990-host\") pod \"node-ca-wtqp4\" (UID: \"79a249fe-49bc-4f12-b1f5-c2e4b3ecc990\") " pod="openshift-image-registry/node-ca-wtqp4"
Oct 31 03:15:26.796691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796700    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796726    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796737    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-bin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796744    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796760    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796764    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-multus-certs\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796774    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796789    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796806    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-kubelet-config\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796824    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796827    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-hostroot\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796840    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796846    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c3f060e6-cd83-4284-98ac-d39a6d8a01be-host-slash\") pod \"iptables-alerter-xrkgp\" (UID: \"c3f060e6-cd83-4284-98ac-d39a6d8a01be\") " pod="openshift-network-operator/iptables-alerter-xrkgp"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796848    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-etc-selinux\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796862    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-host\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796856    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796895    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c81c0c55-4184-4b1b-a431-268e608e3b60-dbus\") pod \"global-pull-secret-syncer-bjxm9\" (UID: \"c81c0c55-4184-4b1b-a431-268e608e3b60\") " pod="kube-system/global-pull-secret-syncer-bjxm9"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796896    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796897    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.796983    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797008    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797031    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797047    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-netns\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797064    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-kubelet\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797080    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-run-ovn-kubernetes\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797054    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797118    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797133    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797150    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797137    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-socket-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797081    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-kubelet\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797203    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797212    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-socket-dir-parent\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797204    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-netns\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797239    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-run\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797260    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797204    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-etc-kubernetes\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797288    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797318    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/b1d993de-0f1b-4b88-a998-344eec8c23ac-hosts-file\") pod \"node-resolver-nnxgp\" (UID: \"b1d993de-0f1b-4b88-a998-344eec8c23ac\") " pod="openshift-dns/node-resolver-nnxgp"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797324    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.797325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797336    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-lib-modules\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797354    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797386    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4e4684e1-6f21-46ca-8dec-b41f06386b69-crio-socket\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797390    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-cnibin\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797412    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797413    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4e4684e1-6f21-46ca-8dec-b41f06386b69-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797436    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.797459    5098 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797463    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.797472    5098 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n: secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797493    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797465    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-run-k8s-cni-cncf-io\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797495    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-run-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.797545    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls podName:97322fdd-f911-47eb-b1b5-c336eb539fbe nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.297526916 +0000 UTC m=+1.822746500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls") pod "image-registry-5fd5dcdd4d-fqp9n" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe") : secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797556    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-sys\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797564    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797587    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797618    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797623    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-multus-conf-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797635    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797651    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797656    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-host-var-lib-cni-multus\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797669    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-registration-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797692    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-etc-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797704    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-device-dir\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797669    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797722    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysconfig\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797734    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797750    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797769    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797769    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-kubernetes\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797804    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797822    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797837    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797852    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/13947923-5437-4f19-b97d-c6e913cf8fc7-sys-fs\") pod \"aws-ebs-csi-driver-node-zxbz7\" (UID: \"13947923-5437-4f19-b97d-c6e913cf8fc7\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zxbz7"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797859    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797903    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/85a63323-7c0f-462b-bdf1-f3a532f2d04c-system-cni-dir\") pod \"multus-additional-cni-plugins-jjjbw\" (UID: \"85a63323-7c0f-462b-bdf1-f3a532f2d04c\") " pod="openshift-multus/multus-additional-cni-plugins-jjjbw"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797906    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-netd\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797906    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797956    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-var-lib-kubelet\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797958    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797907    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-cnibin\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797976    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-sysctl-conf\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.797989    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798021    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.798023    5098 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798055    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.798071    5098 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-859c7cc5c7-8q95x: secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798087    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798094    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/092c4d2f-ae5c-424d-b458-d19bf1e0e8f6-system-cni-dir\") pod \"multus-7p689\" (UID: \"092c4d2f-ae5c-424d-b458-d19bf1e0e8f6\") " pod="openshift-multus/multus-7p689"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798060    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-log-socket\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.798057    5098 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798116    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.798130    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls podName:074d5926-982b-4fcc-913c-682b38ab3683 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.298115433 +0000 UTC m=+1.823335001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls") pod "image-registry-859c7cc5c7-8q95x" (UID: "074d5926-982b-4fcc-913c-682b38ab3683") : secret "image-registry-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:26.798172    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:27.298156676 +0000 UTC m=+1.823376261 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798173    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-systemd-units\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798188    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798218    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798242    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798264    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798269    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f3b2dbb0-3454-4290-81ef-3cf35b21c1d0-etc-systemd\") pod \"tuned-jmgbj\" (UID: \"f3b2dbb0-3454-4290-81ef-3cf35b21c1d0\") " pod="openshift-cluster-node-tuning-operator/tuned-jmgbj"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798295    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-var-lib-openvswitch\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798340    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-host-cni-bin\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.798714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.798345    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/478f1160-d546-4149-aee6-7a3f6f3f520f-node-log\") pod \"ovnkube-node-5fgwn\" (UID: \"478f1160-d546-4149-aee6-7a3f6f3f520f\") " pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:26.800677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.799983    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/231e395b-1984-48a0-865b-b9b05c489eec-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-f49fv\" (UID: \"231e395b-1984-48a0-865b-b9b05c489eec\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.800677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.800022    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9a4b3f3c-5b83-481b-9038-aad4508357bf-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-4927g\" (UID: \"9a4b3f3c-5b83-481b-9038-aad4508357bf\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.800677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.800388    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40423a9f-278f-4f36-87b1-2d05afa56fa7-metrics-certs\") pod \"network-metrics-daemon-bmxnp\" (UID: \"40423a9f-278f-4f36-87b1-2d05afa56fa7\") " pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:26.872028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.871946    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"
Oct 31 03:15:26.872198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.872044    5098 scope.go:122] "RemoveContainer" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"
Oct 31 03:15:26.874362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.874343    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"
Oct 31 03:15:26.879243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.879224    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g"
Oct 31 03:15:26.882427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:26.882406    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bmxnp"
Oct 31 03:15:27.165304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.165222    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n"]
Oct 31 03:15:27.168154 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:27.168115    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod378f839e_767e_421d_b8c7_65369e94a937.slice/crio-6ca35da69653607452d4deec27124a8774dc063954d8b8560d853218af76de03 WatchSource:0}: Error finding container 6ca35da69653607452d4deec27124a8774dc063954d8b8560d853218af76de03: Status 404 returned error can't find the container with id 6ca35da69653607452d4deec27124a8774dc063954d8b8560d853218af76de03
Oct 31 03:15:27.169701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.169677    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv"]
Oct 31 03:15:27.170360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.170339    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:15:27.201594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.201563    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:27.201796 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.201774    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:28.201757588 +0000 UTC m=+2.726977155 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:27.214592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.214558    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-4927g"]
Oct 31 03:15:27.215276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.215258    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bmxnp"]
Oct 31 03:15:27.216761 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:27.216738    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40423a9f_278f_4f36_87b1_2d05afa56fa7.slice/crio-218bab4bb99963b231632c6f8da663b51ba3f8ea0a93738cf5f5af80dfad1f9c WatchSource:0}: Error finding container 218bab4bb99963b231632c6f8da663b51ba3f8ea0a93738cf5f5af80dfad1f9c: Status 404 returned error can't find the container with id 218bab4bb99963b231632c6f8da663b51ba3f8ea0a93738cf5f5af80dfad1f9c
Oct 31 03:15:27.217327 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:27.217300    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4b3f3c_5b83_481b_9038_aad4508357bf.slice/crio-7eeeccf5ea14fce439da842d2a1686ed54c7c4371a4456275ea086951e2a3837 WatchSource:0}: Error finding container 7eeeccf5ea14fce439da842d2a1686ed54c7c4371a4456275ea086951e2a3837: Status 404 returned error can't find the container with id 7eeeccf5ea14fce439da842d2a1686ed54c7c4371a4456275ea086951e2a3837
Oct 31 03:15:27.302734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.302700    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:27.302898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.302810    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:27.302898 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.302853    5098 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:27.302898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.302866    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:27.302898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.302894    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:27.303095 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.302926    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:28.302911558 +0000 UTC m=+2.828131126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:27.303095 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.303000    5098 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:27.303095 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.303054    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:28.303039674 +0000 UTC m=+2.828259247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:27.305361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.305338    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"image-registry-5fd5dcdd4d-fqp9n\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") " pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:27.305468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.305375    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"image-registry-859c7cc5c7-8q95x\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") " pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:27.473803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.473726    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:27.491916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.491888    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:27.677442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.677405    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g" event={"ID":"9a4b3f3c-5b83-481b-9038-aad4508357bf","Type":"ContainerStarted","Data":"7eeeccf5ea14fce439da842d2a1686ed54c7c4371a4456275ea086951e2a3837"}
Oct 31 03:15:27.678865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.678837    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n" event={"ID":"378f839e-767e-421d-b8c7-65369e94a937","Type":"ContainerStarted","Data":"6ca35da69653607452d4deec27124a8774dc063954d8b8560d853218af76de03"}
Oct 31 03:15:27.680246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.680230    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:15:27.680668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.680653    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/1.log"
Oct 31 03:15:27.680751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.680684    5098 generic.go:356] "Generic (PLEG): container finished" podID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerID="e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403" exitCode=255
Oct 31 03:15:27.680751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.680747    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" event={"ID":"1b4b412a-ef91-4c3e-89c3-9287445d404c","Type":"ContainerDied","Data":"e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403"}
Oct 31 03:15:27.681234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.680796    5098 scope.go:122] "RemoveContainer" containerID="660cf08736eae79cc1ccbcca9bb84af2075028e1478f6a6c3bca810bf7bf7421"
Oct 31 03:15:27.681234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.681153    5098 scope.go:122] "RemoveContainer" containerID="e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403"
Oct 31 03:15:27.681429 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:27.681408    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:27.682353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.682332    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv" event={"ID":"231e395b-1984-48a0-865b-b9b05c489eec","Type":"ContainerStarted","Data":"709bc9dcb5b1623e70c06246d2f57bc30aa6eab9b7e216ef530b9e04463ccd9c"}
Oct 31 03:15:27.687517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.684145    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmxnp" event={"ID":"40423a9f-278f-4f36-87b1-2d05afa56fa7","Type":"ContainerStarted","Data":"218bab4bb99963b231632c6f8da663b51ba3f8ea0a93738cf5f5af80dfad1f9c"}
Oct 31 03:15:27.753851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.753763    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:15:27.757516 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:27.757483    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97322fdd_f911_47eb_b1b5_c336eb539fbe.slice/crio-54329aed2cefeb9cd3020f303a2202b90683b924b8660fbf52276d5b8c90d05b WatchSource:0}: Error finding container 54329aed2cefeb9cd3020f303a2202b90683b924b8660fbf52276d5b8c90d05b: Status 404 returned error can't find the container with id 54329aed2cefeb9cd3020f303a2202b90683b924b8660fbf52276d5b8c90d05b
Oct 31 03:15:27.783939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:27.783908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:15:27.795116 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:27.795076    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod074d5926_982b_4fcc_913c_682b38ab3683.slice/crio-ebaf37a951dd0e0b46e67ed632a61a40a04472e0c4cebf2d23f18f77113c68dc WatchSource:0}: Error finding container ebaf37a951dd0e0b46e67ed632a61a40a04472e0c4cebf2d23f18f77113c68dc: Status 404 returned error can't find the container with id ebaf37a951dd0e0b46e67ed632a61a40a04472e0c4cebf2d23f18f77113c68dc
Oct 31 03:15:28.228092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.228043    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:28.228380 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:28.228357    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:30.228333762 +0000 UTC m=+4.753553334 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:28.333405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.332891    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:28.333405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.332942    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:28.333405 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:28.333076    5098 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:28.333405 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:28.333128    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:30.333116316 +0000 UTC m=+4.858335896 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:28.333853 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:28.333827    5098 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:28.333919 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:28.333907    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:30.333889083 +0000 UTC m=+4.859108671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:28.694925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.694752    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:15:28.700236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.700195    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" event={"ID":"97322fdd-f911-47eb-b1b5-c336eb539fbe","Type":"ContainerStarted","Data":"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"}
Oct 31 03:15:28.700405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.700267    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" event={"ID":"97322fdd-f911-47eb-b1b5-c336eb539fbe","Type":"ContainerStarted","Data":"54329aed2cefeb9cd3020f303a2202b90683b924b8660fbf52276d5b8c90d05b"}
Oct 31 03:15:28.703168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.703125    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x" event={"ID":"074d5926-982b-4fcc-913c-682b38ab3683","Type":"ContainerStarted","Data":"85f0f710f129ed8ed4651c87c582c9a156010fd60905e75f4c99383736835a25"}
Oct 31 03:15:28.703370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.703183    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x" event={"ID":"074d5926-982b-4fcc-913c-682b38ab3683","Type":"ContainerStarted","Data":"ebaf37a951dd0e0b46e67ed632a61a40a04472e0c4cebf2d23f18f77113c68dc"}
Oct 31 03:15:28.703370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:28.703300    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:29.711456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.711410    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv" event={"ID":"231e395b-1984-48a0-865b-b9b05c489eec","Type":"ContainerStarted","Data":"a31a435df7238ed5d80b4ff512c932150895c5655f074624df6f0ba058531f54"}
Oct 31 03:15:29.713288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.713256    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmxnp" event={"ID":"40423a9f-278f-4f36-87b1-2d05afa56fa7","Type":"ContainerStarted","Data":"be944353114b99f05c76df6cc352aa10bb33cbcdf7addd9104d806403677bfde"}
Oct 31 03:15:29.713403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.713307    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bmxnp" event={"ID":"40423a9f-278f-4f36-87b1-2d05afa56fa7","Type":"ContainerStarted","Data":"bd7e642d36e59a8f3c7a821e79294c6ff6a1ce8655cb9bfb83f6b3c5fccb23cd"}
Oct 31 03:15:29.715162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.715130    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-4927g" event={"ID":"9a4b3f3c-5b83-481b-9038-aad4508357bf","Type":"ContainerStarted","Data":"75f2d0f7d087ebf6d6c89e0b4544080b49a721de87287ce03e0cb440518c30aa"}
Oct 31 03:15:29.717134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.717096    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-2cg4n" event={"ID":"378f839e-767e-421d-b8c7-65369e94a937","Type":"ContainerStarted","Data":"be497696d52b6aa26832622bf780e400b0ed43da10b3dc09c9b11a9d45ca1024"}
Oct 31 03:15:29.717564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.717541    5098 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:15:29.869672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.869580    5098 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:29.870080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:29.870065    5098 scope.go:122] "RemoveContainer" containerID="e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403"
Oct 31 03:15:29.870335 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:29.870308    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:30.254497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.254396    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:30.254666 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:30.254530    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle podName:b91cefe7-2e73-4f49-8d39-1187617dc044 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:34.254516917 +0000 UTC m=+8.779736485 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle") pod "router-default-5497575d9c-lm8wr" (UID: "b91cefe7-2e73-4f49-8d39-1187617dc044") : configmap references non-existent config key: service-ca.crt
Oct 31 03:15:30.355727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.355683    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:30.355913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.355734    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:30.355913 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:30.355841    5098 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 03:15:30.355913 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:30.355847    5098 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:30.355913 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:30.355895    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls podName:4d7fc633-6f9f-4d5a-adfc-394293b6e4af nodeName:}" failed. No retries permitted until 2026-10-31 03:15:34.355882282 +0000 UTC m=+8.881101853 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls") pod "dns-default-lbdjh" (UID: "4d7fc633-6f9f-4d5a-adfc-394293b6e4af") : secret "dns-default-metrics-tls" not found
Oct 31 03:15:30.356045 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:30.355928    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls podName:4e4684e1-6f21-46ca-8dec-b41f06386b69 nodeName:}" failed. No retries permitted until 2026-10-31 03:15:34.355911968 +0000 UTC m=+8.881131534 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4crls" (UID: "4e4684e1-6f21-46ca-8dec-b41f06386b69") : secret "insights-runtime-extractor-tls" not found
Oct 31 03:15:30.719115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.719081    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:30.722522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.722493    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-f49fv" event={"ID":"231e395b-1984-48a0-865b-b9b05c489eec","Type":"ContainerStarted","Data":"74fe0f9b72afc1ce6ea37be8ed068a24d19bb71f9dfe60c445f8e96ba0d19b7e"}
Oct 31 03:15:30.723997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.723973    5098 generic.go:356] "Generic (PLEG): container finished" podID="aa4f9f2d-e0c9-46f2-8100-06f434f66322" containerID="370e937027cfa282cca915ad1f8231a69a10f117ae89459f5430f5c3acf3bbde" exitCode=0
Oct 31 03:15:30.724118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.724062    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms" event={"ID":"aa4f9f2d-e0c9-46f2-8100-06f434f66322","Type":"ContainerDied","Data":"370e937027cfa282cca915ad1f8231a69a10f117ae89459f5430f5c3acf3bbde"}
Oct 31 03:15:30.724375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.724360    5098 scope.go:122] "RemoveContainer" containerID="370e937027cfa282cca915ad1f8231a69a10f117ae89459f5430f5c3acf3bbde"
Oct 31 03:15:30.975850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:30.975766    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:15:31.729964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:31.729919    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-w6tms" event={"ID":"aa4f9f2d-e0c9-46f2-8100-06f434f66322","Type":"ContainerStarted","Data":"fa7b31f1afa399b8354a5ed67ad3e0e5eea7a5380b2f57c0f731c7a40f44db15"}
Oct 31 03:15:33.608564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:33.608537    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:34.290968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.290925    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:34.291551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.291531    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b91cefe7-2e73-4f49-8d39-1187617dc044-service-ca-bundle\") pod \"router-default-5497575d9c-lm8wr\" (UID: \"b91cefe7-2e73-4f49-8d39-1187617dc044\") " pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:34.374042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.374003    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:34.392313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.392285    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:34.392468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.392378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:34.394689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.394661    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4e4684e1-6f21-46ca-8dec-b41f06386b69-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4crls\" (UID: \"4e4684e1-6f21-46ca-8dec-b41f06386b69\") " pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:34.394782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.394708    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/4d7fc633-6f9f-4d5a-adfc-394293b6e4af-metrics-tls\") pod \"dns-default-lbdjh\" (UID: \"4d7fc633-6f9f-4d5a-adfc-394293b6e4af\") " pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:34.395580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.395566    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4crls"
Oct 31 03:15:34.658152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.658128    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5497575d9c-lm8wr"]
Oct 31 03:15:34.660539 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:34.660510    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91cefe7_2e73_4f49_8d39_1187617dc044.slice/crio-da340e85523bfd9b69ec0bdb1b71c752cfcf7826044d47b1d0896b248eebdd3a WatchSource:0}: Error finding container da340e85523bfd9b69ec0bdb1b71c752cfcf7826044d47b1d0896b248eebdd3a: Status 404 returned error can't find the container with id da340e85523bfd9b69ec0bdb1b71c752cfcf7826044d47b1d0896b248eebdd3a
Oct 31 03:15:34.679998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.679973    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4crls"]
Oct 31 03:15:34.693512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.693472    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:34.693781 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:34.693747    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e4684e1_6f21_46ca_8dec_b41f06386b69.slice/crio-9ce482f1255928b83a4243b7908a49312622256a245a87a41cd45fc1daf14dee WatchSource:0}: Error finding container 9ce482f1255928b83a4243b7908a49312622256a245a87a41cd45fc1daf14dee: Status 404 returned error can't find the container with id 9ce482f1255928b83a4243b7908a49312622256a245a87a41cd45fc1daf14dee
Oct 31 03:15:34.750659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.750543    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4crls" event={"ID":"4e4684e1-6f21-46ca-8dec-b41f06386b69","Type":"ContainerStarted","Data":"0838d629c415318db1d518aa690eb4170bd83b3fb03441a7c6702bcc4c8cd896"}
Oct 31 03:15:34.755640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.755314    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4crls" event={"ID":"4e4684e1-6f21-46ca-8dec-b41f06386b69","Type":"ContainerStarted","Data":"9ce482f1255928b83a4243b7908a49312622256a245a87a41cd45fc1daf14dee"}
Oct 31 03:15:34.755640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.755373    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5497575d9c-lm8wr" event={"ID":"b91cefe7-2e73-4f49-8d39-1187617dc044","Type":"ContainerStarted","Data":"91c9b86d7481bfa568968a971b32cb7d9de4b4934ce705bb143402cce1e14ced"}
Oct 31 03:15:34.755640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.755390    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5497575d9c-lm8wr" event={"ID":"b91cefe7-2e73-4f49-8d39-1187617dc044","Type":"ContainerStarted","Data":"da340e85523bfd9b69ec0bdb1b71c752cfcf7826044d47b1d0896b248eebdd3a"}
Oct 31 03:15:34.842990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.842960    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:34.843257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.843239    5098 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:15:34.978985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:34.978959    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lbdjh"]
Oct 31 03:15:34.982831 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:34.982797    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7fc633_6f9f_4d5a_adfc_394293b6e4af.slice/crio-7841ee0e747cba33a82c4b8c80f77d2a48077d3d48ddf1dda964bb9d07136633 WatchSource:0}: Error finding container 7841ee0e747cba33a82c4b8c80f77d2a48077d3d48ddf1dda964bb9d07136633: Status 404 returned error can't find the container with id 7841ee0e747cba33a82c4b8c80f77d2a48077d3d48ddf1dda964bb9d07136633
Oct 31 03:15:35.374844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:35.374806    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:35.377447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:35.377421    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:35.761455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:35.761409    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4crls" event={"ID":"4e4684e1-6f21-46ca-8dec-b41f06386b69","Type":"ContainerStarted","Data":"8eab4e0b2fdbd38cb24527013e86d47158360c40e6b51581b04129496db9c946"}
Oct 31 03:15:35.762737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:35.762705    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbdjh" event={"ID":"4d7fc633-6f9f-4d5a-adfc-394293b6e4af","Type":"ContainerStarted","Data":"7841ee0e747cba33a82c4b8c80f77d2a48077d3d48ddf1dda964bb9d07136633"}
Oct 31 03:15:36.145997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:36.145954    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:36.146419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:36.146404    5098 scope.go:122] "RemoveContainer" containerID="e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403"
Oct 31 03:15:36.146638 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:15:36.146601    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-nq5jp_openshift-console-operator(1b4b412a-ef91-4c3e-89c3-9287445d404c)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c"
Oct 31 03:15:36.767804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:36.767772    5098 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:15:37.604525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.604443    5098 scope.go:122] "RemoveContainer" containerID="e5e023398c7a7101327d9d40efc5e64ea73d805e26b0916e87754fbc9fb63403"
Oct 31 03:15:37.775026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.774998    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:15:37.775480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.775121    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" event={"ID":"1b4b412a-ef91-4c3e-89c3-9287445d404c","Type":"ContainerStarted","Data":"31ac69f46260ff5e501860daf4b7d5d238a6e934b67216081101dcb14456bde4"}
Oct 31 03:15:37.775480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.775435    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:37.777270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.777240    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbdjh" event={"ID":"4d7fc633-6f9f-4d5a-adfc-394293b6e4af","Type":"ContainerStarted","Data":"3dc983a5d4ce066c03fdd10e8348f89aa39c62ca7f510dd8ff644c8352dc30d0"}
Oct 31 03:15:37.777401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.777287    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lbdjh" event={"ID":"4d7fc633-6f9f-4d5a-adfc-394293b6e4af","Type":"ContainerStarted","Data":"4aea357a6dad7630b794842baaca28431241f208343eec8de2aa6fd5cb8ed383"}
Oct 31 03:15:37.777401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.777315    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:37.779389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.779359    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4crls" event={"ID":"4e4684e1-6f21-46ca-8dec-b41f06386b69","Type":"ContainerStarted","Data":"23108839b5c50335fa485823b186a1520bd2917f0a6a32a564be948d100528a8"}
Oct 31 03:15:37.981137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:37.981049    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp"
Oct 31 03:15:39.651112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:39.651070    5098 patch_prober.go:29] interesting pod/image-registry-5fd5dcdd4d-fqp9n container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Oct 31 03:15:39.651495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:39.651130    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:15:40.313776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.313740    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-mv884"]
Oct 31 03:15:40.318670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.318599    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"]
Oct 31 03:15:40.318819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.318729    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.322086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.322066    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:40.326208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.326179    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:15:40.328087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.328048    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-shd2g\""
Oct 31 03:15:40.334232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.334207    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 03:15:40.339715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.339686    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mv884"]
Oct 31 03:15:40.349442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.349410    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 03:15:40.386572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.386539    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"]
Oct 31 03:15:40.421923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.421887    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-56fb5f95b9-s9pcp"]
Oct 31 03:15:40.425238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.425221    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.426241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.426220    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5qvfw\""
Oct 31 03:15:40.426304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.426220    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 03:15:40.435100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.435077    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:40.436346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.436326    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5497575d9c-lm8wr"
Oct 31 03:15:40.445705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.445679    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89lf7\" (UniqueName: \"kubernetes.io/projected/54e44444-9207-4253-aaa5-77b849d81581-kube-api-access-89lf7\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.445822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.445751    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d7ff043-ea26-4c96-874d-c2d6ab18b6a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jnt8\" (UID: \"5d7ff043-ea26-4c96-874d-c2d6ab18b6a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:40.445858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.445824    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e44444-9207-4253-aaa5-77b849d81581-tmp\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.452917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.452885    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56fb5f95b9-s9pcp"]
Oct 31 03:15:40.546448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546415    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-image-registry-private-configuration\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.546448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546451    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-bound-sa-token\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.546650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546479    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e44444-9207-4253-aaa5-77b849d81581-tmp\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.546715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546701    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5xmr\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-kube-api-access-t5xmr\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.546781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546751    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-tls\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.546897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546873    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f6f81f6-dd4e-460e-9f61-b09e7300702e-ca-trust-extracted\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.547031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546907    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-installation-pull-secrets\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.547031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546947    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89lf7\" (UniqueName: \"kubernetes.io/projected/54e44444-9207-4253-aaa5-77b849d81581-kube-api-access-89lf7\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.547031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.546955    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/54e44444-9207-4253-aaa5-77b849d81581-tmp\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.547172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.547095    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-trusted-ca\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.547172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.547135    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d7ff043-ea26-4c96-874d-c2d6ab18b6a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jnt8\" (UID: \"5d7ff043-ea26-4c96-874d-c2d6ab18b6a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:40.547172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.547158    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-certificates\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.549449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.549411    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/5d7ff043-ea26-4c96-874d-c2d6ab18b6a9-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-6jnt8\" (UID: \"5d7ff043-ea26-4c96-874d-c2d6ab18b6a9\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:40.558597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.558571    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89lf7\" (UniqueName: \"kubernetes.io/projected/54e44444-9207-4253-aaa5-77b849d81581-kube-api-access-89lf7\") pod \"downloads-6dd789c8b7-mv884\" (UID: \"54e44444-9207-4253-aaa5-77b849d81581\") " pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.636756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.636676    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:40.644047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.644017    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:40.648032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.647997    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-trusted-ca\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648187    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-certificates\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648299    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-image-registry-private-configuration\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648407    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-bound-sa-token\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648538    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5xmr\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-kube-api-access-t5xmr\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648643    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-tls\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648744    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f6f81f6-dd4e-460e-9f61-b09e7300702e-ca-trust-extracted\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.648852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.648838    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-installation-pull-secrets\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.652641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.652241    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-installation-pull-secrets\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.652641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.652581    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f6f81f6-dd4e-460e-9f61-b09e7300702e-ca-trust-extracted\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.653270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.652846    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-certificates\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.653990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.653965    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f6f81f6-dd4e-460e-9f61-b09e7300702e-trusted-ca\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.655306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.655282    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f6f81f6-dd4e-460e-9f61-b09e7300702e-image-registry-private-configuration\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.656427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.656401    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-registry-tls\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.662467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.662438    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-bound-sa-token\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.662767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.662745    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5xmr\" (UniqueName: \"kubernetes.io/projected/5f6f81f6-dd4e-460e-9f61-b09e7300702e-kube-api-access-t5xmr\") pod \"image-registry-56fb5f95b9-s9pcp\" (UID: \"5f6f81f6-dd4e-460e-9f61-b09e7300702e\") " pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.737396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.737364    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:40.920438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.920413    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-mv884"]
Oct 31 03:15:40.922870 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:40.922846    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e44444_9207_4253_aaa5_77b849d81581.slice/crio-b71f4f0a4a33680445f7b76ed447a214bda2df179a1f2592eddc9a0588e4ff91 WatchSource:0}: Error finding container b71f4f0a4a33680445f7b76ed447a214bda2df179a1f2592eddc9a0588e4ff91: Status 404 returned error can't find the container with id b71f4f0a4a33680445f7b76ed447a214bda2df179a1f2592eddc9a0588e4ff91
Oct 31 03:15:40.932750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:40.932690    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"]
Oct 31 03:15:40.934887 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:40.934852    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d7ff043_ea26_4c96_874d_c2d6ab18b6a9.slice/crio-78d0ce0884e77f0decc55f7d31768733545f4f0e5204efe818ee838703722091 WatchSource:0}: Error finding container 78d0ce0884e77f0decc55f7d31768733545f4f0e5204efe818ee838703722091: Status 404 returned error can't find the container with id 78d0ce0884e77f0decc55f7d31768733545f4f0e5204efe818ee838703722091
Oct 31 03:15:41.050677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.050632    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-56fb5f95b9-s9pcp"]
Oct 31 03:15:41.056454 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:41.056418    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6f81f6_dd4e_460e_9f61_b09e7300702e.slice/crio-d579b3cc714629d48dd054fb50c6a7cc055da44f75630085f9a8cdfaff99afea WatchSource:0}: Error finding container d579b3cc714629d48dd054fb50c6a7cc055da44f75630085f9a8cdfaff99afea: Status 404 returned error can't find the container with id d579b3cc714629d48dd054fb50c6a7cc055da44f75630085f9a8cdfaff99afea
Oct 31 03:15:41.798438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.798380    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8" event={"ID":"5d7ff043-ea26-4c96-874d-c2d6ab18b6a9","Type":"ContainerStarted","Data":"78d0ce0884e77f0decc55f7d31768733545f4f0e5204efe818ee838703722091"}
Oct 31 03:15:41.800382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.800346    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp" event={"ID":"5f6f81f6-dd4e-460e-9f61-b09e7300702e","Type":"ContainerStarted","Data":"b40d42ca32396b44906484ae9424f4f8f5b85e178ea56783e8289f6d8e491297"}
Oct 31 03:15:41.800532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.800404    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp" event={"ID":"5f6f81f6-dd4e-460e-9f61-b09e7300702e","Type":"ContainerStarted","Data":"d579b3cc714629d48dd054fb50c6a7cc055da44f75630085f9a8cdfaff99afea"}
Oct 31 03:15:41.800532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.800478    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:15:41.801712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.801680    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mv884" event={"ID":"54e44444-9207-4253-aaa5-77b849d81581","Type":"ContainerStarted","Data":"b71f4f0a4a33680445f7b76ed447a214bda2df179a1f2592eddc9a0588e4ff91"}
Oct 31 03:15:41.825439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:41.825375    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp" podStartSLOduration=1.825358868 podStartE2EDuration="1.825358868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:41.822899481 +0000 UTC m=+16.348119075" watchObservedRunningTime="2026-10-31 03:15:41.825358868 +0000 UTC m=+16.350578489"
Oct 31 03:15:42.808745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:42.808708    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8" event={"ID":"5d7ff043-ea26-4c96-874d-c2d6ab18b6a9","Type":"ContainerStarted","Data":"97e3feae70ce9c8f0409a394679b6adca3ecf08e463d509a4814aa9d2fd9568b"}
Oct 31 03:15:42.809301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:42.808907    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:42.814224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:42.814184    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8"
Oct 31 03:15:42.827134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:42.827079    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8" podStartSLOduration=1.72318798 podStartE2EDuration="2.82706222s" totalImagesPullingTime="1.10387424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:42.825206117 +0000 UTC m=+17.350425706" watchObservedRunningTime="2026-10-31 03:15:42.82706222 +0000 UTC m=+17.352281809"
Oct 31 03:15:44.365906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:44.365871    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6csjw"
Oct 31 03:15:46.423413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:46.423378    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5fgwn"
Oct 31 03:15:47.786093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:47.786061    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lbdjh"
Oct 31 03:15:48.871358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.871326    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9lkzg"]
Oct 31 03:15:48.877895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.877858    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:48.916407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.916367    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 03:15:48.916701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.916539    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 03:15:48.968860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.968820    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 03:15:48.969051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.968916    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 03:15:48.969105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:48.969054    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p2pq6\""
Oct 31 03:15:49.037569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037520    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-root\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037573    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037719    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-metrics-client-ca\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037794    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-sys\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037813    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-tls\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037902    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.037969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.037939    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-textfile\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.038065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.038013    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-wtmp\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.038065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.038046    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdgj\" (UniqueName: \"kubernetes.io/projected/a0ac836a-6fac-414b-886c-99fee2b253d2-kube-api-access-gxdgj\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139233    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-root\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139275    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139318    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-metrics-client-ca\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139355    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-sys\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-tls\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139430    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139461    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-textfile\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139489    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-wtmp\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139513    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdgj\" (UniqueName: \"kubernetes.io/projected/a0ac836a-6fac-414b-886c-99fee2b253d2-kube-api-access-gxdgj\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.139913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.139896    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-root\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.140575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.140374    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-sys\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.140575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.140510    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-wtmp\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.140927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.140899    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-accelerators-collector-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.141016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.140954    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-textfile\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.141214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.141193    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a0ac836a-6fac-414b-886c-99fee2b253d2-metrics-client-ca\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.142963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.142941    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-tls\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.149297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.149275    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdgj\" (UniqueName: \"kubernetes.io/projected/a0ac836a-6fac-414b-886c-99fee2b253d2-kube-api-access-gxdgj\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.152202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.152181    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a0ac836a-6fac-414b-886c-99fee2b253d2-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9lkzg\" (UID: \"a0ac836a-6fac-414b-886c-99fee2b253d2\") " pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.200366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.200310    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9lkzg"
Oct 31 03:15:49.215566 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:15:49.215528    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ac836a_6fac_414b_886c_99fee2b253d2.slice/crio-6c3cf695026d7c42124c08ec50ab480b56f0087dc66d822fdea56fa4d5c52678 WatchSource:0}: Error finding container 6c3cf695026d7c42124c08ec50ab480b56f0087dc66d822fdea56fa4d5c52678: Status 404 returned error can't find the container with id 6c3cf695026d7c42124c08ec50ab480b56f0087dc66d822fdea56fa4d5c52678
Oct 31 03:15:49.725013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.724984    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:15:49.870097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:49.870056    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lkzg" event={"ID":"a0ac836a-6fac-414b-886c-99fee2b253d2","Type":"ContainerStarted","Data":"6c3cf695026d7c42124c08ec50ab480b56f0087dc66d822fdea56fa4d5c52678"}
Oct 31 03:15:50.332702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:50.332671    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:15:50.876453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:50.876401    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lkzg" event={"ID":"a0ac836a-6fac-414b-886c-99fee2b253d2","Type":"ContainerStarted","Data":"1b63384d2421e29ebb27b0e4a9c0ee81e908c7af6fc07117dac4b03d5bd522dc"}
Oct 31 03:15:51.882687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:51.882651    5098 generic.go:356] "Generic (PLEG): container finished" podID="a0ac836a-6fac-414b-886c-99fee2b253d2" containerID="1b63384d2421e29ebb27b0e4a9c0ee81e908c7af6fc07117dac4b03d5bd522dc" exitCode=0
Oct 31 03:15:51.883171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:51.882716    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lkzg" event={"ID":"a0ac836a-6fac-414b-886c-99fee2b253d2","Type":"ContainerDied","Data":"1b63384d2421e29ebb27b0e4a9c0ee81e908c7af6fc07117dac4b03d5bd522dc"}
Oct 31 03:15:52.890214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:52.890176    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lkzg" event={"ID":"a0ac836a-6fac-414b-886c-99fee2b253d2","Type":"ContainerStarted","Data":"cfc2e159e5c266d191ef5210916d508c573ec8be9fcd84dc946cb3bc7ee1506c"}
Oct 31 03:15:52.890727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:52.890264    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9lkzg" event={"ID":"a0ac836a-6fac-414b-886c-99fee2b253d2","Type":"ContainerStarted","Data":"4612f7d4df918127c51703e486d606761769eb300f2ddeb3bb99e9448b9196c9"}
Oct 31 03:15:52.913097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:52.913035    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9lkzg" podStartSLOduration=3.33737808 podStartE2EDuration="4.913019702s" totalImagesPullingTime="1.499371766s" totalInitContainerRuntime="76.269856ms" isStatefulPod=true podCreationTimestamp="2026-10-31 03:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:52.912432246 +0000 UTC m=+27.437651837" watchObservedRunningTime="2026-10-31 03:15:52.913019702 +0000 UTC m=+27.438239291"
Oct 31 03:15:57.916483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:57.916441    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-mv884" event={"ID":"54e44444-9207-4253-aaa5-77b849d81581","Type":"ContainerStarted","Data":"545479b95a801b7aa063b014717cf52ff7541754c4be005fbd36e01bb54888f1"}
Oct 31 03:15:57.917979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:57.917692    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:57.933710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:57.933541    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-mv884"
Oct 31 03:15:57.941300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:15:57.941238    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-mv884" podStartSLOduration=1.394882197 podStartE2EDuration="17.941218852s" totalImagesPullingTime="16.546336655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:15:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:15:57.939157688 +0000 UTC m=+32.464377277" watchObservedRunningTime="2026-10-31 03:15:57.941218852 +0000 UTC m=+32.466438441"
Oct 31 03:16:00.979494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:00.979465    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-4ww6v"
Oct 31 03:16:01.075179 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:16:01.075144    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:16:02.814692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:02.814652    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-56fb5f95b9-s9pcp"
Oct 31 03:16:02.907963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:02.907925    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:16:05.354802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.354671    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerName="registry" containerID="cri-o://4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15" gracePeriod=30
Oct 31 03:16:05.648164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.648128    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:16:05.688006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.687967    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688019    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688071    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688104    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688133    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688177    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688216    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.688457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.688304    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") pod \"97322fdd-f911-47eb-b1b5-c336eb539fbe\" (UID: \"97322fdd-f911-47eb-b1b5-c336eb539fbe\") "
Oct 31 03:16:05.691140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.691075    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:05.691963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.691907    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:16:05.692131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.692096    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:05.692622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.692586    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:05.693045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.693010    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "kube-api-access-78kpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:05.693164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.693103    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:05.699325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.699137    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:16:05.710163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.710092    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration" pod "97322fdd-f911-47eb-b1b5-c336eb539fbe" (UID: "97322fdd-f911-47eb-b1b5-c336eb539fbe"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:16:05.789783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789747    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78kpw\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-kube-api-access-78kpw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.789783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789783    5098 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-trusted-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789796    5098 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-image-registry-private-configuration\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789811    5098 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789825    5098 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/97322fdd-f911-47eb-b1b5-c336eb539fbe-ca-trust-extracted\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789838    5098 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/97322fdd-f911-47eb-b1b5-c336eb539fbe-installation-pull-secrets\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789850    5098 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/97322fdd-f911-47eb-b1b5-c336eb539fbe-bound-sa-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.790003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.789862    5098 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/97322fdd-f911-47eb-b1b5-c336eb539fbe-registry-certificates\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:05.960885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.960789    5098 generic.go:356] "Generic (PLEG): container finished" podID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerID="4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15" exitCode=0
Oct 31 03:16:05.960885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.960870    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"
Oct 31 03:16:05.961195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.960872    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" event={"ID":"97322fdd-f911-47eb-b1b5-c336eb539fbe","Type":"ContainerDied","Data":"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"}
Oct 31 03:16:05.961195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.960982    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n" event={"ID":"97322fdd-f911-47eb-b1b5-c336eb539fbe","Type":"ContainerDied","Data":"54329aed2cefeb9cd3020f303a2202b90683b924b8660fbf52276d5b8c90d05b"}
Oct 31 03:16:05.961195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.961012    5098 scope.go:122] "RemoveContainer" containerID="4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"
Oct 31 03:16:05.972097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.972071    5098 scope.go:122] "RemoveContainer" containerID="4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"
Oct 31 03:16:05.972486 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:16:05.972462    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15\": container with ID starting with 4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15 not found: ID does not exist" containerID="4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"
Oct 31 03:16:05.972552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.972498    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15"} err="failed to get container status \"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15\": rpc error: code = NotFound desc = could not find container \"4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15\": container with ID starting with 4bae46c8ef2b61b4f73ea393db43e01e54122086cc18455d85745b21bec83b15 not found: ID does not exist"
Oct 31 03:16:05.998387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:05.998349    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:16:06.003579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:06.003549    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5fd5dcdd4d-fqp9n"]
Oct 31 03:16:06.167752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:06.167709    5098 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6csjw_6d23fdfb-3765-488e-bad2-d80a7fe3a162/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:06.353063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:06.353014    5098 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6csjw_6d23fdfb-3765-488e-bad2-d80a7fe3a162/konnectivity-agent/0.log" limit=1024
Oct 31 03:16:07.609236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:07.609199    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" path="/var/lib/kubelet/pods/97322fdd-f911-47eb-b1b5-c336eb539fbe/volumes"
Oct 31 03:16:27.932238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:27.932179    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x" podUID="074d5926-982b-4fcc-913c-682b38ab3683" containerName="registry" containerID="cri-o://85f0f710f129ed8ed4651c87c582c9a156010fd60905e75f4c99383736835a25" gracePeriod=30
Oct 31 03:16:28.057787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.057755    5098 generic.go:356] "Generic (PLEG): container finished" podID="074d5926-982b-4fcc-913c-682b38ab3683" containerID="85f0f710f129ed8ed4651c87c582c9a156010fd60905e75f4c99383736835a25" exitCode=0
Oct 31 03:16:28.057939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.057831    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x" event={"ID":"074d5926-982b-4fcc-913c-682b38ab3683","Type":"ContainerDied","Data":"85f0f710f129ed8ed4651c87c582c9a156010fd60905e75f4c99383736835a25"}
Oct 31 03:16:28.227679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.227650    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:16:28.285808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285769    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.285987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285815    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.285987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285850    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.285987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285883    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.285987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285910    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.285987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.285939    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.286254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.286060    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.286254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.286099    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") pod \"074d5926-982b-4fcc-913c-682b38ab3683\" (UID: \"074d5926-982b-4fcc-913c-682b38ab3683\") "
Oct 31 03:16:28.286702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.286262    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:28.286702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.286385    5098 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-trusted-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.286702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.286660    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:28.288581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.288551    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:16:28.288764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.288728    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:16:28.288884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.288772    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:28.288942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.288894    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "kube-api-access-dfn8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:28.289092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.289071    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:28.295326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.295291    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted" pod "074d5926-982b-4fcc-913c-682b38ab3683" (UID: "074d5926-982b-4fcc-913c-682b38ab3683"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:16:28.387197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387159    5098 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-installation-pull-secrets\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387192    5098 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-bound-sa-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387203    5098 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/074d5926-982b-4fcc-913c-682b38ab3683-ca-trust-extracted\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387212    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfn8t\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-kube-api-access-dfn8t\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387221    5098 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/074d5926-982b-4fcc-913c-682b38ab3683-registry-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387231    5098 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/074d5926-982b-4fcc-913c-682b38ab3683-image-registry-private-configuration\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:28.387446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:28.387239    5098 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/074d5926-982b-4fcc-913c-682b38ab3683-registry-certificates\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:29.064485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.064441    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x" event={"ID":"074d5926-982b-4fcc-913c-682b38ab3683","Type":"ContainerDied","Data":"ebaf37a951dd0e0b46e67ed632a61a40a04472e0c4cebf2d23f18f77113c68dc"}
Oct 31 03:16:29.064485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.064472    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-859c7cc5c7-8q95x"
Oct 31 03:16:29.064959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.064500    5098 scope.go:122] "RemoveContainer" containerID="85f0f710f129ed8ed4651c87c582c9a156010fd60905e75f4c99383736835a25"
Oct 31 03:16:29.087918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.087886    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:16:29.091766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.091733    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-859c7cc5c7-8q95x"]
Oct 31 03:16:29.609671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:29.609636    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="074d5926-982b-4fcc-913c-682b38ab3683" path="/var/lib/kubelet/pods/074d5926-982b-4fcc-913c-682b38ab3683/volumes"
Oct 31 03:16:31.056314 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:16:31.056272    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:16:31.075531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:31.075498    5098 generic.go:356] "Generic (PLEG): container finished" podID="61a64191-2d4c-48e4-bf08-e010b704cda5" containerID="d255b1d9a28f402d451aaba83f68d0445b5091fcb668a88099d83d0169930526" exitCode=0
Oct 31 03:16:31.075727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:31.075570    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f" event={"ID":"61a64191-2d4c-48e4-bf08-e010b704cda5","Type":"ContainerDied","Data":"d255b1d9a28f402d451aaba83f68d0445b5091fcb668a88099d83d0169930526"}
Oct 31 03:16:31.075966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:31.075949    5098 scope.go:122] "RemoveContainer" containerID="d255b1d9a28f402d451aaba83f68d0445b5091fcb668a88099d83d0169930526"
Oct 31 03:16:32.081530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:32.081495    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-vwf4f" event={"ID":"61a64191-2d4c-48e4-bf08-e010b704cda5","Type":"ContainerStarted","Data":"9fcc7233724be298d7cbc4b36b7d51237d4664e0973ccca8185760f0bd316637"}
Oct 31 03:16:41.122688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.122644    5098 generic.go:356] "Generic (PLEG): container finished" podID="bc41b4e8-af8b-420c-960d-8f948efbf80a" containerID="d71266a14902c3149842c1e03b7407d414e2cf93dfc32167b15f428329d9a5cd" exitCode=0
Oct 31 03:16:41.123183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.122711    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j2gqr" event={"ID":"bc41b4e8-af8b-420c-960d-8f948efbf80a","Type":"ContainerDied","Data":"d71266a14902c3149842c1e03b7407d414e2cf93dfc32167b15f428329d9a5cd"}
Oct 31 03:16:41.123183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.123117    5098 scope.go:122] "RemoveContainer" containerID="d71266a14902c3149842c1e03b7407d414e2cf93dfc32167b15f428329d9a5cd"
Oct 31 03:16:41.621236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.621203    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:16:41.621409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.621377    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-278hz-d9lkp" podUID="b34e0666-5012-40e2-a7b3-7f6e4470545a" containerName="insights-gathering" containerID="cri-o://bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792" gracePeriod=30
Oct 31 03:16:41.709185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.709142    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-xks58-zljvf"]
Oct 31 03:16:41.709854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.709830    5098 cpu_manager.go:414] "containerMap: removing container" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerName="registry"
Oct 31 03:16:41.710031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.710015    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerName="registry"
Oct 31 03:16:41.710535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.710517    5098 cpu_manager.go:414] "containerMap: removing container" podUID="074d5926-982b-4fcc-913c-682b38ab3683" containerName="registry"
Oct 31 03:16:41.710682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.710668    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="074d5926-982b-4fcc-913c-682b38ab3683" containerName="registry"
Oct 31 03:16:41.711007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.710991    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="97322fdd-f911-47eb-b1b5-c336eb539fbe" containerName="registry"
Oct 31 03:16:41.711104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.711095    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="074d5926-982b-4fcc-913c-682b38ab3683" containerName="registry"
Oct 31 03:16:41.727882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.727853    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xks58-zljvf"]
Oct 31 03:16:41.728043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.727966    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:41.907187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.907153    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:41.907187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.907181    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:41.907411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.907206    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:41.907411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.907348    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrcfx\" (UniqueName: \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:41.919227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:41.919207    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:16:42.007891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.007851    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") pod \"b34e0666-5012-40e2-a7b3-7f6e4470545a\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") "
Oct 31 03:16:42.008085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.007914    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") pod \"b34e0666-5012-40e2-a7b3-7f6e4470545a\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") "
Oct 31 03:16:42.008085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.007971    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") pod \"b34e0666-5012-40e2-a7b3-7f6e4470545a\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") "
Oct 31 03:16:42.008085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008004    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") pod \"b34e0666-5012-40e2-a7b3-7f6e4470545a\" (UID: \"b34e0666-5012-40e2-a7b3-7f6e4470545a\") "
Oct 31 03:16:42.008244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008159    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.008244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008190    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.008244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008230    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.008396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008280    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrcfx\" (UniqueName: \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.008396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008302    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle" pod "b34e0666-5012-40e2-a7b3-7f6e4470545a" (UID: "b34e0666-5012-40e2-a7b3-7f6e4470545a"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:42.008396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008326    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle" pod "b34e0666-5012-40e2-a7b3-7f6e4470545a" (UID: "b34e0666-5012-40e2-a7b3-7f6e4470545a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:16:42.008584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008399    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path" pod "b34e0666-5012-40e2-a7b3-7f6e4470545a" (UID: "b34e0666-5012-40e2-a7b3-7f6e4470545a"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:16:42.008584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008421    5098 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-trusted-ca-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:42.008584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.008443    5098 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b34e0666-5012-40e2-a7b3-7f6e4470545a-service-ca-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:42.009047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.009017    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.009175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.009049    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.009242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.009215    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.010526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.010499    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd" pod "b34e0666-5012-40e2-a7b3-7f6e4470545a" (UID: "b34e0666-5012-40e2-a7b3-7f6e4470545a"). InnerVolumeSpecName "kube-api-access-99jcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:16:42.018648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.018628    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrcfx\" (UniqueName: \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\") pod \"periodic-gathering-xks58-zljvf\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") " pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.069818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.069784    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:16:42.111146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.111116    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99jcd\" (UniqueName: \"kubernetes.io/projected/b34e0666-5012-40e2-a7b3-7f6e4470545a-kube-api-access-99jcd\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:42.111265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.111149    5098 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b34e0666-5012-40e2-a7b3-7f6e4470545a-archives-path\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:16:42.129012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.128969    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-j2gqr" event={"ID":"bc41b4e8-af8b-420c-960d-8f948efbf80a","Type":"ContainerStarted","Data":"069031b27bebdf4eade5b773bc21db0cb048ccab517990ce73d3b0f6d0c632c8"}
Oct 31 03:16:42.130407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.130372    5098 generic.go:356] "Generic (PLEG): container finished" podID="b34e0666-5012-40e2-a7b3-7f6e4470545a" containerID="bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792" exitCode=2
Oct 31 03:16:42.130407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.130405    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-278hz-d9lkp"
Oct 31 03:16:42.130578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.130431    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-278hz-d9lkp" event={"ID":"b34e0666-5012-40e2-a7b3-7f6e4470545a","Type":"ContainerDied","Data":"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"}
Oct 31 03:16:42.130578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.130480    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-278hz-d9lkp" event={"ID":"b34e0666-5012-40e2-a7b3-7f6e4470545a","Type":"ContainerDied","Data":"ad82f53d7801d1f3a834f1752901839c671b766505637411b3ac11a28f427a81"}
Oct 31 03:16:42.130578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.130508    5098 scope.go:122] "RemoveContainer" containerID="bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"
Oct 31 03:16:42.143797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.143767    5098 scope.go:122] "RemoveContainer" containerID="bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"
Oct 31 03:16:42.144189 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:16:42.144159    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792\": container with ID starting with bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792 not found: ID does not exist" containerID="bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"
Oct 31 03:16:42.144262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.144204    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792"} err="failed to get container status \"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792\": rpc error: code = NotFound desc = could not find container \"bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792\": container with ID starting with bed11566e086c3b516800a9a27cd66e188b61695d3c5517ebe90c0e1f2ad9792 not found: ID does not exist"
Oct 31 03:16:42.169638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.167875    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:16:42.175582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.175551    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-278hz-d9lkp"]
Oct 31 03:16:42.372100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.372072    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-xks58-zljvf"]
Oct 31 03:16:42.374501 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:16:42.374459    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbddb03_ff9c_4eb1_b8de_b983c9889372.slice/crio-614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a WatchSource:0}: Error finding container 614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a: Status 404 returned error can't find the container with id 614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a
Oct 31 03:16:42.610434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.610402    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5497575d9c-lm8wr_b91cefe7-2e73-4f49-8d39-1187617dc044/router/0.log"
Oct 31 03:16:42.617878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:42.617851    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ft2z2_de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/serve-healthcheck-canary/0.log"
Oct 31 03:16:43.137794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:43.137762    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xks58-zljvf" event={"ID":"1fbddb03-ff9c-4eb1-b8de-b983c9889372","Type":"ContainerStarted","Data":"c7b1bac510cdb6ede7c698e7e6b611d3e36ef30964f83c4194e5435cea60fa04"}
Oct 31 03:16:43.138244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:43.137813    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xks58-zljvf" event={"ID":"1fbddb03-ff9c-4eb1-b8de-b983c9889372","Type":"ContainerStarted","Data":"614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a"}
Oct 31 03:16:43.155133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:43.155084    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-xks58-zljvf" podStartSLOduration=2.155069151 podStartE2EDuration="2.155069151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:16:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:16:43.153559652 +0000 UTC m=+77.678779241" watchObservedRunningTime="2026-10-31 03:16:43.155069151 +0000 UTC m=+77.680288740"
Oct 31 03:16:43.610161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:16:43.610124    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34e0666-5012-40e2-a7b3-7f6e4470545a" path="/var/lib/kubelet/pods/b34e0666-5012-40e2-a7b3-7f6e4470545a/volumes"
Oct 31 03:17:01.056208 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:17:01.056168    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:17:31.056165 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:17:31.056125    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:18:01.056020 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:18:01.055940    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:18:07.484948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:07.484909    5098 generic.go:356] "Generic (PLEG): container finished" podID="1fbddb03-ff9c-4eb1-b8de-b983c9889372" containerID="c7b1bac510cdb6ede7c698e7e6b611d3e36ef30964f83c4194e5435cea60fa04" exitCode=0
Oct 31 03:18:07.484948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:07.484953    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xks58-zljvf" event={"ID":"1fbddb03-ff9c-4eb1-b8de-b983c9889372","Type":"ContainerDied","Data":"c7b1bac510cdb6ede7c698e7e6b611d3e36ef30964f83c4194e5435cea60fa04"}
Oct 31 03:18:08.616439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.616406    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:18:08.717710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.717670    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\") pod \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") "
Oct 31 03:18:08.717920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.717741    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\") pod \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") "
Oct 31 03:18:08.717920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.717757    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\") pod \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") "
Oct 31 03:18:08.717920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.717833    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\" (UniqueName: \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\") pod \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\" (UID: \"1fbddb03-ff9c-4eb1-b8de-b983c9889372\") "
Oct 31 03:18:08.718197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.718164    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle" pod "1fbddb03-ff9c-4eb1-b8de-b983c9889372" (UID: "1fbddb03-ff9c-4eb1-b8de-b983c9889372"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:08.718280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.718191    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle" pod "1fbddb03-ff9c-4eb1-b8de-b983c9889372" (UID: "1fbddb03-ff9c-4eb1-b8de-b983c9889372"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:18:08.718280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.718165    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path" pod "1fbddb03-ff9c-4eb1-b8de-b983c9889372" (UID: "1fbddb03-ff9c-4eb1-b8de-b983c9889372"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:18:08.720058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.720034    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx" pod "1fbddb03-ff9c-4eb1-b8de-b983c9889372" (UID: "1fbddb03-ff9c-4eb1-b8de-b983c9889372"). InnerVolumeSpecName "kube-api-access-lrcfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:18:08.818719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.818678    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrcfx\" (UniqueName: \"kubernetes.io/projected/1fbddb03-ff9c-4eb1-b8de-b983c9889372-kube-api-access-lrcfx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:08.818719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.818712    5098 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-trusted-ca-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:08.818719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.818724    5098 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/1fbddb03-ff9c-4eb1-b8de-b983c9889372-archives-path\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:08.818974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:08.818734    5098 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1fbddb03-ff9c-4eb1-b8de-b983c9889372-service-ca-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:18:09.494559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:09.494530    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-xks58-zljvf"
Oct 31 03:18:09.494744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:09.494562    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-xks58-zljvf" event={"ID":"1fbddb03-ff9c-4eb1-b8de-b983c9889372","Type":"ContainerDied","Data":"614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a"}
Oct 31 03:18:09.494744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:18:09.494622    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614f359bae36c3e66bf474a343bf2f53791d6c927a8f0b0f956d72640c9f127a"
Oct 31 03:18:31.056677 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:18:31.056635    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:19:01.055833 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:19:01.055774    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:19:31.056292 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:19:31.056251    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:20:01.056288 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:20:01.056245    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:20:10.935531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.935495    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"]
Oct 31 03:20:10.936041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.935911    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b34e0666-5012-40e2-a7b3-7f6e4470545a" containerName="insights-gathering"
Oct 31 03:20:10.936041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.935930    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34e0666-5012-40e2-a7b3-7f6e4470545a" containerName="insights-gathering"
Oct 31 03:20:10.936041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.935958    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1fbddb03-ff9c-4eb1-b8de-b983c9889372" containerName="insights-gathering"
Oct 31 03:20:10.936041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.935963    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fbddb03-ff9c-4eb1-b8de-b983c9889372" containerName="insights-gathering"
Oct 31 03:20:10.936234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.936042    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34e0666-5012-40e2-a7b3-7f6e4470545a" containerName="insights-gathering"
Oct 31 03:20:10.936234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.936059    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fbddb03-ff9c-4eb1-b8de-b983c9889372" containerName="insights-gathering"
Oct 31 03:20:10.939620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.939587    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:10.953017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.952986    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"]
Oct 31 03:20:10.970104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.970071    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 03:20:10.970104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:10.970082    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-cdlxs\""
Oct 31 03:20:11.088753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088714    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.088753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088755    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088781    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088803    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088954    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.088997    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afa466d-1d28-4af9-a622-b3be1061dd07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.089071    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.089139    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.089302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.089211    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjb6j\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-kube-api-access-zjb6j\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190416    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190459    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190480    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjb6j\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-kube-api-access-zjb6j\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190511    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190531    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190553    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190570    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.190812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190602    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190887    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afa466d-1d28-4af9-a622-b3be1061dd07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190912    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.190943    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.191062    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.191077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.191538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.191516    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3afa466d-1d28-4af9-a622-b3be1061dd07-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.193003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.192972    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.193244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.193227    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.200805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.200785    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.201078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.201061    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjb6j\" (UniqueName: \"kubernetes.io/projected/3afa466d-1d28-4af9-a622-b3be1061dd07-kube-api-access-zjb6j\") pod \"openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg\" (UID: \"3afa466d-1d28-4af9-a622-b3be1061dd07\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.258068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.258027    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:11.541573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.541520    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"]
Oct 31 03:20:11.543854 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:11.543819    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3afa466d_1d28_4af9_a622_b3be1061dd07.slice/crio-84f937bdaf1be80b92f4f8c67a2e6e224cfd531284c8ad141ba334c5315eed3c WatchSource:0}: Error finding container 84f937bdaf1be80b92f4f8c67a2e6e224cfd531284c8ad141ba334c5315eed3c: Status 404 returned error can't find the container with id 84f937bdaf1be80b92f4f8c67a2e6e224cfd531284c8ad141ba334c5315eed3c
Oct 31 03:20:11.988351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:11.988263    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg" event={"ID":"3afa466d-1d28-4af9-a622-b3be1061dd07","Type":"ContainerStarted","Data":"84f937bdaf1be80b92f4f8c67a2e6e224cfd531284c8ad141ba334c5315eed3c"}
Oct 31 03:20:12.973140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.973096    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"]
Oct 31 03:20:12.977211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.977180    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:12.983652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.983600    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 31 03:20:12.985291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.985245    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 31 03:20:12.985291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.985290    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-kjqfs\""
Oct 31 03:20:12.987520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:12.987268    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"]
Oct 31 03:20:13.000210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.000171    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"]
Oct 31 03:20:13.004219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.004194    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.012353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.012321    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"]
Oct 31 03:20:13.037213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.037168    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"]
Oct 31 03:20:13.041831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.041804    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.048879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.048806    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"]
Oct 31 03:20:13.064064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.064029    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"]
Oct 31 03:20:13.075000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.074962    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.078876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.078845    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"]
Oct 31 03:20:13.083875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.083848    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-5q2bp\""
Oct 31 03:20:13.084096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.083904    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ssqh5\""
Oct 31 03:20:13.084198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.083996    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-vgth7\""
Oct 31 03:20:13.109202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109167    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8z9d\" (UniqueName: \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109234    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109267    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgnxw\" (UniqueName: \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109294    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109321    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgdvw\" (UniqueName: \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109349    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109370    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.109392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109390    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.109800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.109421    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.210625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210566    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8z9d\" (UniqueName: \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.210825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210677    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.210825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210722    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgnxw\" (UniqueName: \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.210825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210752    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.210825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210780    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgdvw\" (UniqueName: \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.210825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210809    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210843    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210870    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210896    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210927    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.210955    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.211094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.211027    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lb2zn\" (UniqueName: \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.211634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.211596    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.211697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.211676    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.211962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.211945    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.212029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.212010    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.212195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.212179    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.212232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.212212    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.220731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.220699    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8z9d\" (UniqueName: \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.220886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.220704    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgdvw\" (UniqueName: \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.220974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.220948    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgnxw\" (UniqueName: \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.299021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.298986    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:13.312425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.312386    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.312557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.312538    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lb2zn\" (UniqueName: \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.312649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.312635    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.312917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.312865    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.313039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.313014    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.322472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.322438    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lb2zn\" (UniqueName: \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.330129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.330095    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:13.367271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.367224    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:13.400659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.400381    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:13.589204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.589169    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"]
Oct 31 03:20:13.596422 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:13.596388    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b66a5d0_7898_4264_8e3d_ef9c7911e4e6.slice/crio-57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7 WatchSource:0}: Error finding container 57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7: Status 404 returned error can't find the container with id 57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7
Oct 31 03:20:13.628842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.628812    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"]
Oct 31 03:20:13.630295 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:13.630268    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fd1415_81c1_4991_9bda_4cdf517ed480.slice/crio-8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745 WatchSource:0}: Error finding container 8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745: Status 404 returned error can't find the container with id 8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745
Oct 31 03:20:13.665250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.665212    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"]
Oct 31 03:20:13.667465 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:13.667427    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89da4a84_c889_4568_bf91_2df9d2ee1b3d.slice/crio-59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6 WatchSource:0}: Error finding container 59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6: Status 404 returned error can't find the container with id 59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6
Oct 31 03:20:13.705258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:13.705220    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"]
Oct 31 03:20:13.709467 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:13.709426    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2323155c_77ea_4d99_8e6b_d7a4cc0859b9.slice/crio-3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5 WatchSource:0}: Error finding container 3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5: Status 404 returned error can't find the container with id 3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5
Oct 31 03:20:14.000729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:14.000645    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8" event={"ID":"89da4a84-c889-4568-bf91-2df9d2ee1b3d","Type":"ContainerStarted","Data":"59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6"}
Oct 31 03:20:14.002031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:14.002000    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerStarted","Data":"3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5"}
Oct 31 03:20:14.003324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:14.003295    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq" event={"ID":"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6","Type":"ContainerStarted","Data":"57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7"}
Oct 31 03:20:14.004946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:14.004912    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6" event={"ID":"38fd1415-81c1-4991-9bda-4cdf517ed480","Type":"ContainerStarted","Data":"8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745"}
Oct 31 03:20:15.715708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:15.715667    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:20:15.716144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:15.715763    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:20:15.716144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:15.715820    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:20:16.025800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:16.025709    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg" event={"ID":"3afa466d-1d28-4af9-a622-b3be1061dd07","Type":"ContainerStarted","Data":"607040b2f77ba15c5774985465a2c380bd6978aa6c15dcaed9ca0e174efed256"}
Oct 31 03:20:16.056722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:16.056662    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg" podStartSLOduration=1.887869469 podStartE2EDuration="6.056643872s" totalImagesPullingTime="4.168774403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:16.055852611 +0000 UTC m=+290.581072203" watchObservedRunningTime="2026-10-31 03:20:16.056643872 +0000 UTC m=+290.581863523"
Oct 31 03:20:16.258425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:16.258382    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:16.266079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:16.266046    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:17.043224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:17.043185    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:17.044235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:17.044205    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg"
Oct 31 03:20:23.085727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:23.085690    5098 generic.go:356] "Generic (PLEG): container finished" podID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerID="22a8b8d6dbe8b4abb42e142b34ee02fe3fa03e409fe274323c36478ffaf1bbb0" exitCode=0
Oct 31 03:20:23.086179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:23.085780    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8" event={"ID":"89da4a84-c889-4568-bf91-2df9d2ee1b3d","Type":"ContainerDied","Data":"22a8b8d6dbe8b4abb42e142b34ee02fe3fa03e409fe274323c36478ffaf1bbb0"}
Oct 31 03:20:23.088561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:23.087928    5098 generic.go:356] "Generic (PLEG): container finished" podID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerID="b27f50f0a21ff1e3c22331146dafec9815ee775f3b67cab193f047cfa15f07b1" exitCode=0
Oct 31 03:20:23.088561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:23.088029    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq" event={"ID":"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6","Type":"ContainerDied","Data":"b27f50f0a21ff1e3c22331146dafec9815ee775f3b67cab193f047cfa15f07b1"}
Oct 31 03:20:24.094227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:24.094189    5098 generic.go:356] "Generic (PLEG): container finished" podID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerID="fa2196e257bf7c3f10d1e79929d1d6b7b0e18a32befccf5b0fe2ac4456134131" exitCode=0
Oct 31 03:20:24.094693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:24.094279    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerDied","Data":"fa2196e257bf7c3f10d1e79929d1d6b7b0e18a32befccf5b0fe2ac4456134131"}
Oct 31 03:20:24.095867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:24.095839    5098 generic.go:356] "Generic (PLEG): container finished" podID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerID="4f5f2dcef39af7bebbf8e73bbc07e4f37ae5ea350daa4ffd1bff108bf96cc7ae" exitCode=0
Oct 31 03:20:24.095981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:24.095870    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6" event={"ID":"38fd1415-81c1-4991-9bda-4cdf517ed480","Type":"ContainerDied","Data":"4f5f2dcef39af7bebbf8e73bbc07e4f37ae5ea350daa4ffd1bff108bf96cc7ae"}
Oct 31 03:20:25.102646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.102595    5098 generic.go:356] "Generic (PLEG): container finished" podID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerID="e49a0c79a71b8c554dcde23fec6c02b0dc667092bc4375e05a5cddfb010c60b0" exitCode=0
Oct 31 03:20:25.103042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.102646    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8" event={"ID":"89da4a84-c889-4568-bf91-2df9d2ee1b3d","Type":"ContainerDied","Data":"e49a0c79a71b8c554dcde23fec6c02b0dc667092bc4375e05a5cddfb010c60b0"}
Oct 31 03:20:25.104777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.104757    5098 generic.go:356] "Generic (PLEG): container finished" podID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerID="1c2ef5776e377d56682bf0b4f47f3b1e5821d1edd5afc5e478705a6f4bd6a99f" exitCode=0
Oct 31 03:20:25.104857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.104810    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerDied","Data":"1c2ef5776e377d56682bf0b4f47f3b1e5821d1edd5afc5e478705a6f4bd6a99f"}
Oct 31 03:20:25.106931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.106909    5098 generic.go:356] "Generic (PLEG): container finished" podID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerID="d3824285aa08bd6e230298e1ba0e7ae21f63712cd732f80590d187eea42d263b" exitCode=0
Oct 31 03:20:25.107029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.107008    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq" event={"ID":"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6","Type":"ContainerDied","Data":"d3824285aa08bd6e230298e1ba0e7ae21f63712cd732f80590d187eea42d263b"}
Oct 31 03:20:25.109656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.109635    5098 generic.go:356] "Generic (PLEG): container finished" podID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerID="1ae1dac37232cb1acf06eeb4cb8da563add990988d3ab322653f1849cf4bc63d" exitCode=0
Oct 31 03:20:25.109742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.109672    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6" event={"ID":"38fd1415-81c1-4991-9bda-4cdf517ed480","Type":"ContainerDied","Data":"1ae1dac37232cb1acf06eeb4cb8da563add990988d3ab322653f1849cf4bc63d"}
Oct 31 03:20:25.639380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.639348    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:20:25.641121 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:25.641097    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:20:31.063945 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:20:31.063867    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:20:35.194062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.194032    5098 generic.go:356] "Generic (PLEG): container finished" podID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerID="759f5bb08c0fdfef01c0270e06b9920e0394e4097991b599b320621f9e05f192" exitCode=0
Oct 31 03:20:35.194473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.194085    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6" event={"ID":"38fd1415-81c1-4991-9bda-4cdf517ed480","Type":"ContainerDied","Data":"759f5bb08c0fdfef01c0270e06b9920e0394e4097991b599b320621f9e05f192"}
Oct 31 03:20:35.196251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.196222    5098 generic.go:356] "Generic (PLEG): container finished" podID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerID="869c2cc2355b325123f952c9bb3dabc34f698dfe2cf8a117a9d73fee0c86041a" exitCode=0
Oct 31 03:20:35.196365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.196303    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8" event={"ID":"89da4a84-c889-4568-bf91-2df9d2ee1b3d","Type":"ContainerDied","Data":"869c2cc2355b325123f952c9bb3dabc34f698dfe2cf8a117a9d73fee0c86041a"}
Oct 31 03:20:35.198594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.198569    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerStarted","Data":"1d112ad12a2b5c13b8c390bc15fc794c85ba4a14e6b9425ce08ab58654482768"}
Oct 31 03:20:35.201049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.200986    5098 generic.go:356] "Generic (PLEG): container finished" podID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerID="e903d8e523e2bca33367ad913cd7c0746f1b922ea8da97221ca21d3c3e04bffb" exitCode=0
Oct 31 03:20:35.201049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.201045    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq" event={"ID":"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6","Type":"ContainerDied","Data":"e903d8e523e2bca33367ad913cd7c0746f1b922ea8da97221ca21d3c3e04bffb"}
Oct 31 03:20:35.264093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:35.264045    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" podStartSLOduration=1.754774309 podStartE2EDuration="22.264026826s" totalImagesPullingTime="20.509252517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:35.263201037 +0000 UTC m=+309.788420641" watchObservedRunningTime="2026-10-31 03:20:35.264026826 +0000 UTC m=+309.789246418"
Oct 31 03:20:36.207495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.207454    5098 generic.go:356] "Generic (PLEG): container finished" podID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerID="1d112ad12a2b5c13b8c390bc15fc794c85ba4a14e6b9425ce08ab58654482768" exitCode=0
Oct 31 03:20:36.207956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.207532    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerDied","Data":"1d112ad12a2b5c13b8c390bc15fc794c85ba4a14e6b9425ce08ab58654482768"}
Oct 31 03:20:36.356278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.356247    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:36.371690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.371667    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:36.395627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.395589    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:36.444353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444314    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\") pod \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") "
Oct 31 03:20:36.444551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444389    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\" (UniqueName: \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\") pod \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") "
Oct 31 03:20:36.444551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444415    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\" (UniqueName: \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\") pod \"38fd1415-81c1-4991-9bda-4cdf517ed480\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") "
Oct 31 03:20:36.444551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444465    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\") pod \"38fd1415-81c1-4991-9bda-4cdf517ed480\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") "
Oct 31 03:20:36.444771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444574    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\") pod \"38fd1415-81c1-4991-9bda-4cdf517ed480\" (UID: \"38fd1415-81c1-4991-9bda-4cdf517ed480\") "
Oct 31 03:20:36.444771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444640    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\") pod \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") "
Oct 31 03:20:36.444771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444679    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\") pod \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\" (UID: \"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6\") "
Oct 31 03:20:36.444771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444721    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\" (UniqueName: \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\") pod \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") "
Oct 31 03:20:36.444973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.444783    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\") pod \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\" (UID: \"89da4a84-c889-4568-bf91-2df9d2ee1b3d\") "
Oct 31 03:20:36.445673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.445065    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle" pod "38fd1415-81c1-4991-9bda-4cdf517ed480" (UID: "38fd1415-81c1-4991-9bda-4cdf517ed480"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.445673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.445594    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle" pod "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" (UID: "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.445857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.445703    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle" pod "89da4a84-c889-4568-bf91-2df9d2ee1b3d" (UID: "89da4a84-c889-4568-bf91-2df9d2ee1b3d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.447301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.447276    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw" pod "38fd1415-81c1-4991-9bda-4cdf517ed480" (UID: "38fd1415-81c1-4991-9bda-4cdf517ed480"). InnerVolumeSpecName "kube-api-access-cgdvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:36.447394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.447307    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw" pod "89da4a84-c889-4568-bf91-2df9d2ee1b3d" (UID: "89da4a84-c889-4568-bf91-2df9d2ee1b3d"). InnerVolumeSpecName "kube-api-access-sgnxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:36.447394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.447315    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d" pod "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" (UID: "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6"). InnerVolumeSpecName "kube-api-access-c8z9d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:36.449411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.449389    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util" pod "89da4a84-c889-4568-bf91-2df9d2ee1b3d" (UID: "89da4a84-c889-4568-bf91-2df9d2ee1b3d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.450517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.450494    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util" pod "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" (UID: "0b66a5d0-7898-4264-8e3d-ef9c7911e4e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.451018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.450996    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util" pod "38fd1415-81c1-4991-9bda-4cdf517ed480" (UID: "38fd1415-81c1-4991-9bda-4cdf517ed480"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:36.546086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546048    5098 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546082    5098 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/38fd1415-81c1-4991-9bda-4cdf517ed480-util\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546093    5098 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546103    5098 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546113    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgnxw\" (UniqueName: \"kubernetes.io/projected/89da4a84-c889-4568-bf91-2df9d2ee1b3d-kube-api-access-sgnxw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546122    5098 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89da4a84-c889-4568-bf91-2df9d2ee1b3d-util\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546130    5098 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-util\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546138    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8z9d\" (UniqueName: \"kubernetes.io/projected/0b66a5d0-7898-4264-8e3d-ef9c7911e4e6-kube-api-access-c8z9d\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:36.546324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:36.546146    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgdvw\" (UniqueName: \"kubernetes.io/projected/38fd1415-81c1-4991-9bda-4cdf517ed480-kube-api-access-cgdvw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:37.214137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.214096    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8"
Oct 31 03:20:37.214588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.214105    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847als2t8" event={"ID":"89da4a84-c889-4568-bf91-2df9d2ee1b3d","Type":"ContainerDied","Data":"59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6"}
Oct 31 03:20:37.214588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.214266    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59c1f7cf889500aeee6e6aef8ae40b0eb150ce731a5bc484ce41a2c36d4191a6"
Oct 31 03:20:37.216092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.216069    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq"
Oct 31 03:20:37.216218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.216085    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cmknhq" event={"ID":"0b66a5d0-7898-4264-8e3d-ef9c7911e4e6","Type":"ContainerDied","Data":"57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7"}
Oct 31 03:20:37.216218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.216125    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d67deceb0293043a42dae0525b488f980eb0c474e803bf155c6c5113fb23b7"
Oct 31 03:20:37.218090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.218067    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6" event={"ID":"38fd1415-81c1-4991-9bda-4cdf517ed480","Type":"ContainerDied","Data":"8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745"}
Oct 31 03:20:37.218185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.218104    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b42badc3eadb403c8b503c55f5148ff0d179110d9f0333ff5f3a07201a27745"
Oct 31 03:20:37.218258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.218225    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6bv2j6"
Oct 31 03:20:37.349045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.349021    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:37.454166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.454131    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\") pod \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") "
Oct 31 03:20:37.454339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.454184    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\" (UniqueName: \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\") pod \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") "
Oct 31 03:20:37.454339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.454235    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\") pod \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\" (UID: \"2323155c-77ea-4d99-8e6b-d7a4cc0859b9\") "
Oct 31 03:20:37.454764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.454737    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle" pod "2323155c-77ea-4d99-8e6b-d7a4cc0859b9" (UID: "2323155c-77ea-4d99-8e6b-d7a4cc0859b9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:37.456337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.456315    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn" pod "2323155c-77ea-4d99-8e6b-d7a4cc0859b9" (UID: "2323155c-77ea-4d99-8e6b-d7a4cc0859b9"). InnerVolumeSpecName "kube-api-access-lb2zn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:20:37.459774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.459749    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util" pod "2323155c-77ea-4d99-8e6b-d7a4cc0859b9" (UID: "2323155c-77ea-4d99-8e6b-d7a4cc0859b9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:20:37.555736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.555702    5098 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-util\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:37.555736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.555733    5098 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-bundle\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:37.555935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:37.555742    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lb2zn\" (UniqueName: \"kubernetes.io/projected/2323155c-77ea-4d99-8e6b-d7a4cc0859b9-kube-api-access-lb2zn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:20:38.225698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:38.225666    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v"
Oct 31 03:20:38.226147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:38.225667    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7tc97v" event={"ID":"2323155c-77ea-4d99-8e6b-d7a4cc0859b9","Type":"ContainerDied","Data":"3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5"}
Oct 31 03:20:38.226147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:38.225809    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d0a8ea8e99d822770ab56e07bc29c14a04c1c37a455ebaca109dc1e111c93d5"
Oct 31 03:20:43.342941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.342909    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"]
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343237    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343249    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343262    5098 cpu_manager.go:414] "containerMap: removing container" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343267    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343276    5098 cpu_manager.go:414] "containerMap: removing container" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343281    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343288    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343292    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="util"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343301    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343305    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343313    5098 cpu_manager.go:414] "containerMap: removing container" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343318    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343323    5098 cpu_manager.go:414] "containerMap: removing container" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343329    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343336    5098 cpu_manager.go:414] "containerMap: removing container" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343340    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343350    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343354    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343364    5098 cpu_manager.go:414] "containerMap: removing container" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343368    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343374    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343378    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="pull"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343386    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343390    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343441    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2323155c-77ea-4d99-8e6b-d7a4cc0859b9" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343449    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b66a5d0-7898-4264-8e3d-ef9c7911e4e6" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343458    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fd1415-81c1-4991-9bda-4cdf517ed480" containerName="extract"
Oct 31 03:20:43.343457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.343464    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="89da4a84-c889-4568-bf91-2df9d2ee1b3d" containerName="extract"
Oct 31 03:20:43.374860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.374823    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"]
Oct 31 03:20:43.375036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.374957    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.406845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.406812    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.406988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.406855    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9fx\" (UniqueName: \"kubernetes.io/projected/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-kube-api-access-dw9fx\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.406988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.406965    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.407072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.406994    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.407072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.407031    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.407137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.407112    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.429739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.429713    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 03:20:43.429885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.429798    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 03:20:43.429885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.429877    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 03:20:43.429982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.429964    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 03:20:43.430393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.430377    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 03:20:43.430465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.430434    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-dw2ct\""
Oct 31 03:20:43.507662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507596    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507693    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507755    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507784    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9fx\" (UniqueName: \"kubernetes.io/projected/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-kube-api-access-dw9fx\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507835    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.507857    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.508143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.508077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-manifests\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.510307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.510271    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-tmp\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.510448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.510332    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-metrics-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.510448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.510357    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-apiservice-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.510448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.510407    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-webhook-cert\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.518837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.518815    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9fx\" (UniqueName: \"kubernetes.io/projected/dc9fd9a3-bd12-4f19-bfa1-3cabdad01085-kube-api-access-dw9fx\") pod \"opendatahub-operator-controller-manager-55975df5cc-4ncnz\" (UID: \"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:43.729830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:43.729733    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:44.039308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:44.039267    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"]
Oct 31 03:20:44.040528 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:20:44.040504    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc9fd9a3_bd12_4f19_bfa1_3cabdad01085.slice/crio-b64d3fbfe2ec9305b6a0681301fe13ff606a75b65a0205711702914636871920 WatchSource:0}: Error finding container b64d3fbfe2ec9305b6a0681301fe13ff606a75b65a0205711702914636871920: Status 404 returned error can't find the container with id b64d3fbfe2ec9305b6a0681301fe13ff606a75b65a0205711702914636871920
Oct 31 03:20:44.042443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:44.042420    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:20:44.254015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:44.253976    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz" event={"ID":"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085","Type":"ContainerStarted","Data":"b64d3fbfe2ec9305b6a0681301fe13ff606a75b65a0205711702914636871920"}
Oct 31 03:20:48.275125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:48.275089    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz" event={"ID":"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085","Type":"ContainerStarted","Data":"c5476446c14c74f63048b9aa591ee37903543e43ca35b19d7634573840cbacf8"}
Oct 31 03:20:49.281673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:49.281639    5098 generic.go:356] "Generic (PLEG): container finished" podID="dc9fd9a3-bd12-4f19-bfa1-3cabdad01085" containerID="c5476446c14c74f63048b9aa591ee37903543e43ca35b19d7634573840cbacf8" exitCode=0
Oct 31 03:20:49.282132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:49.281699    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz" event={"ID":"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085","Type":"ContainerDied","Data":"c5476446c14c74f63048b9aa591ee37903543e43ca35b19d7634573840cbacf8"}
Oct 31 03:20:50.288010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:50.287968    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz" event={"ID":"dc9fd9a3-bd12-4f19-bfa1-3cabdad01085","Type":"ContainerStarted","Data":"7caf4cb2e57c418edcbf95718b5a0da353869d022215d3e19e76413bf1d79e5b"}
Oct 31 03:20:50.288405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:50.288133    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:20:50.336261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:20:50.336209    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz" podStartSLOduration=2.5935026519999997 podStartE2EDuration="7.336194241s" totalImagesPullingTime="4.163121046s" totalInitContainerRuntime="579.570543ms" isStatefulPod=false podCreationTimestamp="2026-10-31 03:20:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:20:50.333940059 +0000 UTC m=+324.859159649" watchObservedRunningTime="2026-10-31 03:20:50.336194241 +0000 UTC m=+324.861413827"
Oct 31 03:21:00.064275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.064234    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"]
Oct 31 03:21:00.070726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.070694    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:00.087271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.087231    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"]
Oct 31 03:21:00.088464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.088434    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 03:21:00.088599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.088522    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Oct 31 03:21:00.088599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.088524    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2k2zq\""
Oct 31 03:21:00.088599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.088552    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 03:21:00.158785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.158746    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dchk\" (UniqueName: \"kubernetes.io/projected/1bd96048-f68d-46b4-be69-75ebc2987909-kube-api-access-7dchk\") pod \"dns-operator-controller-manager-8557cbdb84-grs9n\" (UID: \"1bd96048-f68d-46b4-be69-75ebc2987909\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:00.259268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.259223    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dchk\" (UniqueName: \"kubernetes.io/projected/1bd96048-f68d-46b4-be69-75ebc2987909-kube-api-access-7dchk\") pod \"dns-operator-controller-manager-8557cbdb84-grs9n\" (UID: \"1bd96048-f68d-46b4-be69-75ebc2987909\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:00.268599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.268566    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dchk\" (UniqueName: \"kubernetes.io/projected/1bd96048-f68d-46b4-be69-75ebc2987909-kube-api-access-7dchk\") pod \"dns-operator-controller-manager-8557cbdb84-grs9n\" (UID: \"1bd96048-f68d-46b4-be69-75ebc2987909\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:00.383454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.383361    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:00.677657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:00.677626    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"]
Oct 31 03:21:00.679588 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:00.679561    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd96048_f68d_46b4_be69_75ebc2987909.slice/crio-cc2f860c9b31511c3a50c694d9fda6ced00b09401dfb435ab03e146b53955667 WatchSource:0}: Error finding container cc2f860c9b31511c3a50c694d9fda6ced00b09401dfb435ab03e146b53955667: Status 404 returned error can't find the container with id cc2f860c9b31511c3a50c694d9fda6ced00b09401dfb435ab03e146b53955667
Oct 31 03:21:01.055909 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:21:01.055868    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:21:01.294259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:01.294230    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-55975df5cc-4ncnz"
Oct 31 03:21:01.349147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:01.349058    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n" event={"ID":"1bd96048-f68d-46b4-be69-75ebc2987909","Type":"ContainerStarted","Data":"cc2f860c9b31511c3a50c694d9fda6ced00b09401dfb435ab03e146b53955667"}
Oct 31 03:21:05.168420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.168382    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b9vrj"]
Oct 31 03:21:05.172597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.172563    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.179000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.178937    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 31 03:21:05.179227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.179191    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 31 03:21:05.179555 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.179533    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-bvkmq\""
Oct 31 03:21:05.180905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.180847    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b9vrj"]
Oct 31 03:21:05.309163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.309117    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-bound-sa-token\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.309350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.309173    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m49k9\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-kube-api-access-m49k9\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.410249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.410219    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-bound-sa-token\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.410506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.410271    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m49k9\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-kube-api-access-m49k9\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.427239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.427172    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-bound-sa-token\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.427413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.427392    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m49k9\" (UniqueName: \"kubernetes.io/projected/e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b-kube-api-access-m49k9\") pod \"cert-manager-597b95cb84-b9vrj\" (UID: \"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b\") " pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.495436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.495397    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b9vrj"
Oct 31 03:21:05.773821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:05.773739    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b9vrj"]
Oct 31 03:21:05.776827 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:05.776783    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7fb438f_bf27_4aeb_91f9_6a7ba6fdbc3b.slice/crio-e2bc410c65577b518e2b4398ab66fd32838277bb7d94c3ea4118a64e596f2286 WatchSource:0}: Error finding container e2bc410c65577b518e2b4398ab66fd32838277bb7d94c3ea4118a64e596f2286: Status 404 returned error can't find the container with id e2bc410c65577b518e2b4398ab66fd32838277bb7d94c3ea4118a64e596f2286
Oct 31 03:21:06.375986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.375948    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n" event={"ID":"1bd96048-f68d-46b4-be69-75ebc2987909","Type":"ContainerStarted","Data":"d9a2c1f5693c3b1e8d390e55dc461c163a7532e7590282bee270c45a3be74532"}
Oct 31 03:21:06.376472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.376009    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:06.377354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.377325    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b9vrj" event={"ID":"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b","Type":"ContainerStarted","Data":"e2bc410c65577b518e2b4398ab66fd32838277bb7d94c3ea4118a64e596f2286"}
Oct 31 03:21:06.395423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.395365    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n" podStartSLOduration=1.680342355 podStartE2EDuration="6.395352719s" totalImagesPullingTime="4.715010364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:06.393968265 +0000 UTC m=+340.919187855" watchObservedRunningTime="2026-10-31 03:21:06.395352719 +0000 UTC m=+340.920572307"
Oct 31 03:21:06.967413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.967369    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"]
Oct 31 03:21:06.971467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.971426    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:06.981306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.981274    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 31 03:21:06.981428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.981370    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 31 03:21:06.984156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.984069    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"]
Oct 31 03:21:06.985966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:06.985940    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-vz5hn\""
Oct 31 03:21:07.126319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.126277    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf285\" (UniqueName: \"kubernetes.io/projected/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-kube-api-access-lf285\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.126519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.126461    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-tmp\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.228370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.227504    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf285\" (UniqueName: \"kubernetes.io/projected/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-kube-api-access-lf285\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.228370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.227577    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-tmp\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.228370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.228021    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-tmp\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.237869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.237810    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf285\" (UniqueName: \"kubernetes.io/projected/1609b99e-20a7-48bb-aa05-6bc5cca1cf51-kube-api-access-lf285\") pod \"jobset-operator-674cd59dc5-qgf6w\" (UID: \"1609b99e-20a7-48bb-aa05-6bc5cca1cf51\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.288417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.288374    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"
Oct 31 03:21:07.604073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.604042    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w"]
Oct 31 03:21:07.606862 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:07.606827    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1609b99e_20a7_48bb_aa05_6bc5cca1cf51.slice/crio-1819cdcc5e877b40a8eae5f1d470cd1ceac2c39bc89684f92b608d8029dd2710 WatchSource:0}: Error finding container 1819cdcc5e877b40a8eae5f1d470cd1ceac2c39bc89684f92b608d8029dd2710: Status 404 returned error can't find the container with id 1819cdcc5e877b40a8eae5f1d470cd1ceac2c39bc89684f92b608d8029dd2710
Oct 31 03:21:07.865665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.865576    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:07.872581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.872552    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:07.876883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.876855    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-msbqv\""
Oct 31 03:21:07.897573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:07.890176    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:08.036357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.036316    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.036553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.036447    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2k7m\" (UniqueName: \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.138094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.138014    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.138262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.138092    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2k7m\" (UniqueName: \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.138408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.138389    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.147682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.147650    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2k7m\" (UniqueName: \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\") pod \"kuadrant-operator-controller-manager-cf8469d7b-rfrtk\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.190457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.190417    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:08.398869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.398788    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w" event={"ID":"1609b99e-20a7-48bb-aa05-6bc5cca1cf51","Type":"ContainerStarted","Data":"1819cdcc5e877b40a8eae5f1d470cd1ceac2c39bc89684f92b608d8029dd2710"}
Oct 31 03:21:08.480647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:08.480600    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:08.482863 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:08.482829    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777a1afa_bbce_4884_9f91_d425760e1b80.slice/crio-f7884841a3298300185c862c364894b7575edfb57f42a7e4c558a262f4930e16 WatchSource:0}: Error finding container f7884841a3298300185c862c364894b7575edfb57f42a7e4c558a262f4930e16: Status 404 returned error can't find the container with id f7884841a3298300185c862c364894b7575edfb57f42a7e4c558a262f4930e16
Oct 31 03:21:09.407895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:09.407854    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" event={"ID":"777a1afa-bbce-4884-9f91-d425760e1b80","Type":"ContainerStarted","Data":"f7884841a3298300185c862c364894b7575edfb57f42a7e4c558a262f4930e16"}
Oct 31 03:21:10.415349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:10.415313    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b9vrj" event={"ID":"e7fb438f-bf27-4aeb-91f9-6a7ba6fdbc3b","Type":"ContainerStarted","Data":"a92c23583ad49d834b845afdfa456e7cbd115eaebca3ee3759514dbf8242e3d5"}
Oct 31 03:21:10.445150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:10.445088    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b9vrj" podStartSLOduration=1.5055352819999999 podStartE2EDuration="5.445069581s" totalImagesPullingTime="3.939534299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:10.442809096 +0000 UTC m=+344.968028686" watchObservedRunningTime="2026-10-31 03:21:10.445069581 +0000 UTC m=+344.970289169"
Oct 31 03:21:12.400134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.400094    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"]
Oct 31 03:21:12.404201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.404171    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.408725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.408687    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k6dbg\""
Oct 31 03:21:12.418436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.418359    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"]
Oct 31 03:21:12.436640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.435016    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w" event={"ID":"1609b99e-20a7-48bb-aa05-6bc5cca1cf51","Type":"ContainerStarted","Data":"f50cfacc8472394c5e4148bc88a2017d37ebb364321d1481ff31c7f55767ed91"}
Oct 31 03:21:12.504361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.504278    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-qgf6w" podStartSLOduration=2.335897201 podStartE2EDuration="6.504257905s" totalImagesPullingTime="4.168360704s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:12.501758499 +0000 UTC m=+347.026978089" watchObservedRunningTime="2026-10-31 03:21:12.504257905 +0000 UTC m=+347.029477497"
Oct 31 03:21:12.591925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.591877    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.591953    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljdf\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-kube-api-access-kljdf\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592124    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6832d278-dba5-4b0d-af12-726ea83e1764-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592175    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592217    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592251    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592296    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592340    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.592376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.592365    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6832d278-dba5-4b0d-af12-726ea83e1764-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693257    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693302    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693321    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6832d278-dba5-4b0d-af12-726ea83e1764-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693357    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693381    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljdf\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-kube-api-access-kljdf\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693426    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6832d278-dba5-4b0d-af12-726ea83e1764-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693458    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693495    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.693734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693524    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.694009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693741    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.694009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.693776    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.694143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.694107    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-data\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.694279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.694147    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.694490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.694463    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6832d278-dba5-4b0d-af12-726ea83e1764-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.700299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.700265    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6832d278-dba5-4b0d-af12-726ea83e1764-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.700502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.700469    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6832d278-dba5-4b0d-af12-726ea83e1764-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.722223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.722191    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-istio-token\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:12.725333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:12.725297    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljdf\" (UniqueName: \"kubernetes.io/projected/6832d278-dba5-4b0d-af12-726ea83e1764-kube-api-access-kljdf\") pod \"data-science-gateway-data-science-gateway-class-649697ffd5czs5l\" (UID: \"6832d278-dba5-4b0d-af12-726ea83e1764\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:13.024433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.024345    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:13.323500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.323468    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"]
Oct 31 03:21:13.325963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:13.325931    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6832d278_dba5_4b0d_af12_726ea83e1764.slice/crio-fd9a9debf0f98e98ab11069255715676d10d9b1e413f531a6b234b4c4de9d929 WatchSource:0}: Error finding container fd9a9debf0f98e98ab11069255715676d10d9b1e413f531a6b234b4c4de9d929: Status 404 returned error can't find the container with id fd9a9debf0f98e98ab11069255715676d10d9b1e413f531a6b234b4c4de9d929
Oct 31 03:21:13.328525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.328484    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:13.328639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.328578    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:13.328686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.328645    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:21:13.444394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.444358    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l" event={"ID":"6832d278-dba5-4b0d-af12-726ea83e1764","Type":"ContainerStarted","Data":"d0edc5fd59926e33cd48b4eff1de9d39b00f3e3aaa2d70e221553e38e77fcacb"}
Oct 31 03:21:13.444826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.444423    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l" event={"ID":"6832d278-dba5-4b0d-af12-726ea83e1764","Type":"ContainerStarted","Data":"fd9a9debf0f98e98ab11069255715676d10d9b1e413f531a6b234b4c4de9d929"}
Oct 31 03:21:13.446774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.446742    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" event={"ID":"777a1afa-bbce-4884-9f91-d425760e1b80","Type":"ContainerStarted","Data":"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"}
Oct 31 03:21:13.446980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.446966    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:13.480775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.480706    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l" podStartSLOduration=1.480683532 podStartE2EDuration="1.480683532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:13.4788842 +0000 UTC m=+348.004103789" watchObservedRunningTime="2026-10-31 03:21:13.480683532 +0000 UTC m=+348.005903122"
Oct 31 03:21:13.522492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.521401    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" podStartSLOduration=2.192748441 podStartE2EDuration="6.521383608s" totalImagesPullingTime="4.328635167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:13.520254971 +0000 UTC m=+348.045474561" watchObservedRunningTime="2026-10-31 03:21:13.521383608 +0000 UTC m=+348.046603220"
Oct 31 03:21:13.735820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.735728    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"]
Oct 31 03:21:13.739595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.739576    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:13.758789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.758754    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"]
Oct 31 03:21:13.843203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.843174    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-t5jd8\""
Oct 31 03:21:13.904713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:13.904666    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7swbk\" (UniqueName: \"kubernetes.io/projected/f07ce24f-e523-48d4-9458-bab5f9b7f2af-kube-api-access-7swbk\") pod \"authorino-operator-7b7bb4bc77-ld4pq\" (UID: \"f07ce24f-e523-48d4-9458-bab5f9b7f2af\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:14.005990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.005894    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7swbk\" (UniqueName: \"kubernetes.io/projected/f07ce24f-e523-48d4-9458-bab5f9b7f2af-kube-api-access-7swbk\") pod \"authorino-operator-7b7bb4bc77-ld4pq\" (UID: \"f07ce24f-e523-48d4-9458-bab5f9b7f2af\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:14.022669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.022632    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7swbk\" (UniqueName: \"kubernetes.io/projected/f07ce24f-e523-48d4-9458-bab5f9b7f2af-kube-api-access-7swbk\") pod \"authorino-operator-7b7bb4bc77-ld4pq\" (UID: \"f07ce24f-e523-48d4-9458-bab5f9b7f2af\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:14.025344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.025326    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:14.030649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.030591    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:14.052820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.052784    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:14.337261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.337149    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"]
Oct 31 03:21:14.339904 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:14.339867    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07ce24f_e523_48d4_9458_bab5f9b7f2af.slice/crio-4792c5664ad157da127782dfda3260414eb6b9a38d9478d2ca4e28d1491f7d32 WatchSource:0}: Error finding container 4792c5664ad157da127782dfda3260414eb6b9a38d9478d2ca4e28d1491f7d32: Status 404 returned error can't find the container with id 4792c5664ad157da127782dfda3260414eb6b9a38d9478d2ca4e28d1491f7d32
Oct 31 03:21:14.454992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.454940    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq" event={"ID":"f07ce24f-e523-48d4-9458-bab5f9b7f2af","Type":"ContainerStarted","Data":"4792c5664ad157da127782dfda3260414eb6b9a38d9478d2ca4e28d1491f7d32"}
Oct 31 03:21:14.455459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.455379    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:14.456206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:14.456183    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-649697ffd5czs5l"
Oct 31 03:21:17.394220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:17.394187    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-grs9n"
Oct 31 03:21:17.471850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:17.471807    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq" event={"ID":"f07ce24f-e523-48d4-9458-bab5f9b7f2af","Type":"ContainerStarted","Data":"db6f58e32678d2e0ce9d4932d9835cc18f62b56cd24d9b1656f3be2912a8dea1"}
Oct 31 03:21:17.472041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:17.471896    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:17.490131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:17.490075    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq" podStartSLOduration=2.210662551 podStartE2EDuration="4.490060886s" totalImagesPullingTime="2.279398335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:17.488960358 +0000 UTC m=+352.014179950" watchObservedRunningTime="2026-10-31 03:21:17.490060886 +0000 UTC m=+352.015280476"
Oct 31 03:21:24.457289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:24.457258    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:28.479868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:28.479829    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-ld4pq"
Oct 31 03:21:31.056683 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:21:31.056637    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:21:40.558801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.558764    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"]
Oct 31 03:21:40.567034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.567005    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.570259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.570225    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"]
Oct 31 03:21:40.572169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.572143    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Oct 31 03:21:40.572337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.572143    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Oct 31 03:21:40.572402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.572207    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-m4s6q\""
Oct 31 03:21:40.572402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.572279    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Oct 31 03:21:40.637514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.637460    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-manager-config\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.637735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.637637    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqvm7\" (UniqueName: \"kubernetes.io/projected/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-kube-api-access-tqvm7\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.637735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.637688    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-cert\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.637851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.637795    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-metrics-certs\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.738326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.738288    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-cert\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.738524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.738372    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-metrics-certs\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.738524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.738424    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-manager-config\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.738664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.738532    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqvm7\" (UniqueName: \"kubernetes.io/projected/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-kube-api-access-tqvm7\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.739270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.739243    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-manager-config\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.741093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.741066    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-metrics-certs\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.741093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.741083    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-cert\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.747356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.747328    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqvm7\" (UniqueName: \"kubernetes.io/projected/d3fa87e8-279f-4c28-ab71-2ed62d6a04ff-kube-api-access-tqvm7\") pod \"jobset-controller-manager-8575547f5f-rqv7m\" (UID: \"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff\") " pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:40.884040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:40.883951    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:41.161713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.161623    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"]
Oct 31 03:21:41.164233 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:21:41.164204    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3fa87e8_279f_4c28_ab71_2ed62d6a04ff.slice/crio-7f594d9d11170f74d9c0bdf310e325887991e8daaaf737344ea6b55091b2e9b3 WatchSource:0}: Error finding container 7f594d9d11170f74d9c0bdf310e325887991e8daaaf737344ea6b55091b2e9b3: Status 404 returned error can't find the container with id 7f594d9d11170f74d9c0bdf310e325887991e8daaaf737344ea6b55091b2e9b3
Oct 31 03:21:41.191009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.190977    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:41.191212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.191175    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" podUID="777a1afa-bbce-4884-9f91-d425760e1b80" containerName="manager" containerID="cri-o://28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5" gracePeriod=10
Oct 31 03:21:41.440799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.440770    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:41.546672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.546635    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\" (UniqueName: \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\") pod \"777a1afa-bbce-4884-9f91-d425760e1b80\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") "
Oct 31 03:21:41.546894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.546740    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\") pod \"777a1afa-bbce-4884-9f91-d425760e1b80\" (UID: \"777a1afa-bbce-4884-9f91-d425760e1b80\") "
Oct 31 03:21:41.547224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.547187    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume" pod "777a1afa-bbce-4884-9f91-d425760e1b80" (UID: "777a1afa-bbce-4884-9f91-d425760e1b80"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:21:41.549052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.549024    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m" pod "777a1afa-bbce-4884-9f91-d425760e1b80" (UID: "777a1afa-bbce-4884-9f91-d425760e1b80"). InnerVolumeSpecName "kube-api-access-n2k7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:21:41.595088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.595050    5098 generic.go:356] "Generic (PLEG): container finished" podID="777a1afa-bbce-4884-9f91-d425760e1b80" containerID="28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5" exitCode=0
Oct 31 03:21:41.595594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.595101    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"
Oct 31 03:21:41.595594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.595143    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" event={"ID":"777a1afa-bbce-4884-9f91-d425760e1b80","Type":"ContainerDied","Data":"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"}
Oct 31 03:21:41.595594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.595219    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk" event={"ID":"777a1afa-bbce-4884-9f91-d425760e1b80","Type":"ContainerDied","Data":"f7884841a3298300185c862c364894b7575edfb57f42a7e4c558a262f4930e16"}
Oct 31 03:21:41.595594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.595253    5098 scope.go:122] "RemoveContainer" containerID="28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"
Oct 31 03:21:41.596739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.596708    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m" event={"ID":"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff","Type":"ContainerStarted","Data":"7f594d9d11170f74d9c0bdf310e325887991e8daaaf737344ea6b55091b2e9b3"}
Oct 31 03:21:41.607409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.607386    5098 scope.go:122] "RemoveContainer" containerID="28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"
Oct 31 03:21:41.607805 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:21:41.607774    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5\": container with ID starting with 28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5 not found: ID does not exist" containerID="28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"
Oct 31 03:21:41.607895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.607819    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5"} err="failed to get container status \"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5\": rpc error: code = NotFound desc = could not find container \"28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5\": container with ID starting with 28b79fc84a163d14b5a88febaf8c0dae2f09ecf1584fc4c9649c7d08fed7f4f5 not found: ID does not exist"
Oct 31 03:21:41.628349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.628313    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:41.641679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.641631    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-rfrtk"]
Oct 31 03:21:41.647736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.647701    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2k7m\" (UniqueName: \"kubernetes.io/projected/777a1afa-bbce-4884-9f91-d425760e1b80-kube-api-access-n2k7m\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:41.647736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:41.647733    5098 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/777a1afa-bbce-4884-9f91-d425760e1b80-extensions-socket-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:21:43.611805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:43.611764    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777a1afa-bbce-4884-9f91-d425760e1b80" path="/var/lib/kubelet/pods/777a1afa-bbce-4884-9f91-d425760e1b80/volumes"
Oct 31 03:21:45.620057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:45.620020    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m" event={"ID":"d3fa87e8-279f-4c28-ab71-2ed62d6a04ff","Type":"ContainerStarted","Data":"f789d720fbc182f62612d05b04bf6929cb3dc39d3f944bf8698082e4a240b312"}
Oct 31 03:21:45.620505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:45.620153    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:45.638467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:45.638415    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m" podStartSLOduration=1.412786103 podStartE2EDuration="5.638400568s" totalImagesPullingTime="4.225614465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:21:45.63655218 +0000 UTC m=+380.161771772" watchObservedRunningTime="2026-10-31 03:21:45.638400568 +0000 UTC m=+380.163620156"
Oct 31 03:21:56.630211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:56.630178    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m"
Oct 31 03:21:59.961188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.961147    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:21:59.961582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.961522    5098 cpu_manager.go:414] "containerMap: removing container" podUID="777a1afa-bbce-4884-9f91-d425760e1b80" containerName="manager"
Oct 31 03:21:59.961582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.961532    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777a1afa-bbce-4884-9f91-d425760e1b80" containerName="manager"
Oct 31 03:21:59.961689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.961640    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="777a1afa-bbce-4884-9f91-d425760e1b80" containerName="manager"
Oct 31 03:21:59.965936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.965918    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:21:59.972770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.972742    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:21:59.973261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.973238    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-fzbgx\""
Oct 31 03:21:59.996593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:21:59.996556    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:22:00.012339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.012306    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 31 03:22:00.118687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.118592    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px7f6\" (UniqueName: \"kubernetes.io/projected/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-kube-api-access-px7f6\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.118869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.118786    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-config-file\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.219756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.219602    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px7f6\" (UniqueName: \"kubernetes.io/projected/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-kube-api-access-px7f6\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.219930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.219766    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-config-file\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.220424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.220403    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-config-file\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.228261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.228234    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px7f6\" (UniqueName: \"kubernetes.io/projected/bd289733-c246-4cb5-a87f-eeb4a8d1ca4f-kube-api-access-px7f6\") pod \"limitador-limitador-7c9b87d574-hbwwx\" (UID: \"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.279683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.279640    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:00.558439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.558413    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:22:00.560813 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:22:00.560778    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd289733_c246_4cb5_a87f_eeb4a8d1ca4f.slice/crio-74d2c2c7f95747ec0fde49f32ea69dee0b2aaf91f17ca2172dd2dd1fd3c016ec WatchSource:0}: Error finding container 74d2c2c7f95747ec0fde49f32ea69dee0b2aaf91f17ca2172dd2dd1fd3c016ec: Status 404 returned error can't find the container with id 74d2c2c7f95747ec0fde49f32ea69dee0b2aaf91f17ca2172dd2dd1fd3c016ec
Oct 31 03:22:00.696567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:00.696516    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx" event={"ID":"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f","Type":"ContainerStarted","Data":"74d2c2c7f95747ec0fde49f32ea69dee0b2aaf91f17ca2172dd2dd1fd3c016ec"}
Oct 31 03:22:01.057887 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:22:01.057848    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:22:02.707898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:02.707860    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx" event={"ID":"bd289733-c246-4cb5-a87f-eeb4a8d1ca4f","Type":"ContainerStarted","Data":"b8ceab3b2774bc2a5ada11406e5a7b89d315ba7e00629a2389a2534804f2a7f8"}
Oct 31 03:22:02.708328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:02.707965    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:02.789281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:02.789225    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx" podStartSLOduration=2.663772562 podStartE2EDuration="3.789211275s" totalImagesPullingTime="1.125438713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:22:02.788964844 +0000 UTC m=+397.314184434" watchObservedRunningTime="2026-10-31 03:22:02.789211275 +0000 UTC m=+397.314430864"
Oct 31 03:22:13.715216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:22:13.715125    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"
Oct 31 03:22:31.055312 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:22:31.055269    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:23:01.056648 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:23:01.056593    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:23:15.090466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:23:15.090425    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:23:31.055836 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:23:31.055793    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:24:01.056504 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:24:01.056422    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:24:20.767522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.767487    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"]
Oct 31 03:24:20.771163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.771138    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.776720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.776697    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 03:24:20.776720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.776697    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Oct 31 03:24:20.776935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.776718    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 03:24:20.776935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.776793    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tkb4s\""
Oct 31 03:24:20.781466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.781439    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"]
Oct 31 03:24:20.875478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875442    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58m6z\" (UniqueName: \"kubernetes.io/projected/5899e70c-2b85-4866-aa96-3a755adfe0f8-kube-api-access-58m6z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.875478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875488    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.875748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875510    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5899e70c-2b85-4866-aa96-3a755adfe0f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.875748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875654    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.875748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875713    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.875872 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.875794    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976713    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976781    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58m6z\" (UniqueName: \"kubernetes.io/projected/5899e70c-2b85-4866-aa96-3a755adfe0f8-kube-api-access-58m6z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976817    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976841    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5899e70c-2b85-4866-aa96-3a755adfe0f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976888    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.976946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.976945    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.977234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.977211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.977301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.977280    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.977347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.977303    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.979237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.979212    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5899e70c-2b85-4866-aa96-3a755adfe0f8-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.979493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.979477    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5899e70c-2b85-4866-aa96-3a755adfe0f8-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:20.985458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:20.985436    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58m6z\" (UniqueName: \"kubernetes.io/projected/5899e70c-2b85-4866-aa96-3a755adfe0f8-kube-api-access-58m6z\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl\" (UID: \"5899e70c-2b85-4866-aa96-3a755adfe0f8\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:21.086443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:21.086401    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:21.260838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:21.260801    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:24:21.363032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:21.363003    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"]
Oct 31 03:24:21.364937 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:24:21.364906    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5899e70c_2b85_4866_aa96_3a755adfe0f8.slice/crio-15ac9b3a0432ddba93ebd71c1de3db76d5d375d3d1f0fcdc2463e30cd57abd50 WatchSource:0}: Error finding container 15ac9b3a0432ddba93ebd71c1de3db76d5d375d3d1f0fcdc2463e30cd57abd50: Status 404 returned error can't find the container with id 15ac9b3a0432ddba93ebd71c1de3db76d5d375d3d1f0fcdc2463e30cd57abd50
Oct 31 03:24:21.379925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:21.379891    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" event={"ID":"5899e70c-2b85-4866-aa96-3a755adfe0f8","Type":"ContainerStarted","Data":"15ac9b3a0432ddba93ebd71c1de3db76d5d375d3d1f0fcdc2463e30cd57abd50"}
Oct 31 03:24:22.675205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:22.675163    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:24:23.390860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:23.390821    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" event={"ID":"5899e70c-2b85-4866-aa96-3a755adfe0f8","Type":"ContainerStarted","Data":"1b4cc91cc311b120b1a80ffd09a67bb2a6df52571a9238f8e19a0f024f50061a"}
Oct 31 03:24:23.410914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:23.410849    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podStartSLOduration=1.960727332 podStartE2EDuration="3.41083459s" totalImagesPullingTime="1.450107258s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:23.409062352 +0000 UTC m=+537.934281942" watchObservedRunningTime="2026-10-31 03:24:23.41083459 +0000 UTC m=+537.936054178"
Oct 31 03:24:24.050744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.050710    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"]
Oct 31 03:24:24.054469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.054443    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.060647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.060623    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Oct 31 03:24:24.066586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.066553    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"]
Oct 31 03:24:24.086696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.086669    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:24.106146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106114    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.106317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106168    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.106317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106232    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.106317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106288    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.106420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106323    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.106456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.106426    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjsmz\" (UniqueName: \"kubernetes.io/projected/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-kube-api-access-pjsmz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207448    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207493    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207520    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207540    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207584    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjsmz\" (UniqueName: \"kubernetes.io/projected/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-kube-api-access-pjsmz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207625    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.207988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.207954    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.208052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.208008    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.208109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.208047    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.210030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.210005    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.210210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.210193    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.216425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.216396    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjsmz\" (UniqueName: \"kubernetes.io/projected/511eac43-6d5d-4d2f-9c1e-ca4c515c9fea-kube-api-access-pjsmz\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln\" (UID: \"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.371919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.371830    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:24.648135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:24.648110    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"]
Oct 31 03:24:24.649998 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:24:24.649968    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod511eac43_6d5d_4d2f_9c1e_ca4c515c9fea.slice/crio-1e424e9794d9476debfe98c631b9d7126ed467854e89b542edeeba588e375572 WatchSource:0}: Error finding container 1e424e9794d9476debfe98c631b9d7126ed467854e89b542edeeba588e375572: Status 404 returned error can't find the container with id 1e424e9794d9476debfe98c631b9d7126ed467854e89b542edeeba588e375572
Oct 31 03:24:25.005975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:25.005945    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:24:25.401936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:25.401899    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" event={"ID":"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea","Type":"ContainerStarted","Data":"d1cc34ba03fc698d612f74c470ff28cf58524c7fa3aa6b662b2816c1c972a9b9"}
Oct 31 03:24:25.402333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:25.401948    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" event={"ID":"511eac43-6d5d-4d2f-9c1e-ca4c515c9fea","Type":"ContainerStarted","Data":"1e424e9794d9476debfe98c631b9d7126ed467854e89b542edeeba588e375572"}
Oct 31 03:24:25.428696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:25.428640    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podStartSLOduration=1.127225019 podStartE2EDuration="1.428624544s" totalImagesPullingTime="301.399525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:25.426735338 +0000 UTC m=+539.951954927" watchObservedRunningTime="2026-10-31 03:24:25.428624544 +0000 UTC m=+539.953844133"
Oct 31 03:24:26.372484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:26.372443    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:27.541361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:27.541328    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:24:28.357246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.357208    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"]
Oct 31 03:24:28.361205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.361180    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.370689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.370662    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 03:24:28.371107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.371081    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"]
Oct 31 03:24:28.447956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.447916    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.448135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.447973    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.448135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.448028    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.448135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.448062    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q94g\" (UniqueName: \"kubernetes.io/projected/17595e19-e8f1-47af-9b70-0b75e8559dde-kube-api-access-5q94g\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.448135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.448089    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17595e19-e8f1-47af-9b70-0b75e8559dde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.448270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.448220    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549493    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549565    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549647    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549682    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549728    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q94g\" (UniqueName: \"kubernetes.io/projected/17595e19-e8f1-47af-9b70-0b75e8559dde-kube-api-access-5q94g\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.549960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.549764    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17595e19-e8f1-47af-9b70-0b75e8559dde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.550218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.550007    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.550218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.550046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.550218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.550118    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.551922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.551902    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17595e19-e8f1-47af-9b70-0b75e8559dde-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.552208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.552191    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17595e19-e8f1-47af-9b70-0b75e8559dde-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.559105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.559077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q94g\" (UniqueName: \"kubernetes.io/projected/17595e19-e8f1-47af-9b70-0b75e8559dde-kube-api-access-5q94g\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-k66s8\" (UID: \"17595e19-e8f1-47af-9b70-0b75e8559dde\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.675643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.675515    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:28.953532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:28.953194    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"]
Oct 31 03:24:29.422091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.422052    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" event={"ID":"17595e19-e8f1-47af-9b70-0b75e8559dde","Type":"ContainerStarted","Data":"db302aa8974d47eb4208ec338c1b05b64cf5bb2ce323de29c70a1360f58a9bd2"}
Oct 31 03:24:29.422272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.422109    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" event={"ID":"17595e19-e8f1-47af-9b70-0b75e8559dde","Type":"ContainerStarted","Data":"9bf1f9a6d42e206a7a4584b9567152e389b8c3318b1faca38880b146022c3431"}
Oct 31 03:24:29.446321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.446265    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podStartSLOduration=1.142314377 podStartE2EDuration="1.446251985s" totalImagesPullingTime="303.937608ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:29.445346796 +0000 UTC m=+543.970566386" watchObservedRunningTime="2026-10-31 03:24:29.446251985 +0000 UTC m=+543.971471623"
Oct 31 03:24:29.675695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.675656    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:29.709711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.709668    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"]
Oct 31 03:24:29.713418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.713397    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.723275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.723254    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Oct 31 03:24:29.725037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.725013    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"]
Oct 31 03:24:29.762106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.762106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762106    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.762328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762133    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tbsw\" (UniqueName: \"kubernetes.io/projected/9c5fe135-1668-41e1-a2d4-42aa4234e43e-kube-api-access-5tbsw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.762328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762291    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.762402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762358    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.762402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.762397    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863344    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863385    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863434    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863451    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863467    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tbsw\" (UniqueName: \"kubernetes.io/projected/9c5fe135-1668-41e1-a2d4-42aa4234e43e-kube-api-access-5tbsw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863529    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863897    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863922    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.863976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.863965    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.865899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.865881    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5fe135-1668-41e1-a2d4-42aa4234e43e-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.866028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.866010    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5fe135-1668-41e1-a2d4-42aa4234e43e-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:29.871748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:29.871724    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tbsw\" (UniqueName: \"kubernetes.io/projected/9c5fe135-1668-41e1-a2d4-42aa4234e43e-kube-api-access-5tbsw\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-8jbq7\" (UID: \"9c5fe135-1668-41e1-a2d4-42aa4234e43e\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:30.027964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:30.027930    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:30.301515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:30.301489    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"]
Oct 31 03:24:30.303529 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:24:30.303502    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5fe135_1668_41e1_a2d4_42aa4234e43e.slice/crio-213ee025d7ad8a1c42ae6215c5647dc323ce9d6f5f56d02c419386733f0432f3 WatchSource:0}: Error finding container 213ee025d7ad8a1c42ae6215c5647dc323ce9d6f5f56d02c419386733f0432f3: Status 404 returned error can't find the container with id 213ee025d7ad8a1c42ae6215c5647dc323ce9d6f5f56d02c419386733f0432f3
Oct 31 03:24:30.428287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:30.428244    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" event={"ID":"9c5fe135-1668-41e1-a2d4-42aa4234e43e","Type":"ContainerStarted","Data":"213ee025d7ad8a1c42ae6215c5647dc323ce9d6f5f56d02c419386733f0432f3"}
Oct 31 03:24:31.056938 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:24:31.056902    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:24:31.086930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.086897    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:31.099655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.099627    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:31.435757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.435646    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" event={"ID":"9c5fe135-1668-41e1-a2d4-42aa4234e43e","Type":"ContainerStarted","Data":"240eb5e217d4b7fb46896d9af4aa59d387eec7c6df07d032ca97e4df8d91127f"}
Oct 31 03:24:31.447115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.447086    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:24:31.457482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.457432    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podStartSLOduration=2.170305973 podStartE2EDuration="2.45738898s" totalImagesPullingTime="287.083007ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:31.456358118 +0000 UTC m=+545.981577709" watchObservedRunningTime="2026-10-31 03:24:31.45738898 +0000 UTC m=+545.982608569"
Oct 31 03:24:31.541549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:31.541514    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:24:32.028353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:32.028313    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:34.372582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:34.372536    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:34.385501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:34.385467    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:34.463351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:34.463317    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:24:38.676687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:38.676649    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:38.689118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:38.689085    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:39.488421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:39.488392    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8"
Oct 31 03:24:40.028746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:40.028708    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:40.041437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:40.041399    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:40.494389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:40.494305    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7"
Oct 31 03:24:41.494304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.494273    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:24:41.497932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.497913    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.502331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.502306    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-tw4cw\""
Oct 31 03:24:41.511911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.511882    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:24:41.580084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580043    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580090    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580196    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580234    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580416    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580438    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580455    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2xwx\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580681    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.580847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.580748    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.681854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.681811    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.681871    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.681929    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.681951    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.681976    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2xwx\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682026    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682042    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682058    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682293    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682439    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682452    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682556    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682684    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.682766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.682751    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.684376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.684357    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.684749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.684732    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.689780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.689753    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.690002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.689983    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2xwx\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\") pod \"maas-default-gateway-openshift-default-769df7846c-lgqnh\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:41.812843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:41.812809    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:42.094127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.094100    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:24:42.095595 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:24:42.095569    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c8bef5_4de6_46dd_9356_8cf4b282cb16.slice/crio-ea7fcc9f9657c876bcc399baa97026512aa51432efdfcf05c414ad51062dd757 WatchSource:0}: Error finding container ea7fcc9f9657c876bcc399baa97026512aa51432efdfcf05c414ad51062dd757: Status 404 returned error can't find the container with id ea7fcc9f9657c876bcc399baa97026512aa51432efdfcf05c414ad51062dd757
Oct 31 03:24:42.098081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.098047    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:42.098181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.098129    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:42.098249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.098177    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:42.506230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.506128    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" event={"ID":"03c8bef5-4de6-46dd-9356-8cf4b282cb16","Type":"ContainerStarted","Data":"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"}
Oct 31 03:24:42.506833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.506793    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" event={"ID":"03c8bef5-4de6-46dd-9356-8cf4b282cb16","Type":"ContainerStarted","Data":"ea7fcc9f9657c876bcc399baa97026512aa51432efdfcf05c414ad51062dd757"}
Oct 31 03:24:42.528501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.528441    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podStartSLOduration=1.528401824 podStartE2EDuration="1.528401824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:42.526087105 +0000 UTC m=+557.051306705" watchObservedRunningTime="2026-10-31 03:24:42.528401824 +0000 UTC m=+557.053621484"
Oct 31 03:24:42.813106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:42.813068    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:43.814860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:43.814823    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:43.815264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:43.814880    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:44.813520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:44.813475    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:44.813734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:44.813555    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:45.813906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:45.813868    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:45.814315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:45.813940    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:46.813694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:46.813653    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:46.813915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:46.813718    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:47.813239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:47.813201    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:47.813768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:47.813273    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:48.813413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:48.813379    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body=
Oct 31 03:24:48.813829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:48.813437    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused"
Oct 31 03:24:50.343434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:50.343381    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:50.343957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:50.343849    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:50.344931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:50.344877    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:24:56.576033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.575997    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:24:56.582468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.582445    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.597718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.597688    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"]
Oct 31 03:24:56.603974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.603949    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:24:56.604314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.604290    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.618881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.618843    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"]
Oct 31 03:24:56.732482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732445    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.732482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732486    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.732733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732506    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.732733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732651    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-kube-api-access-qqkzj\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.732799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732730    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.732799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732753    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.732866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732809    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee37c72-f7bb-459c-9f06-df7401cc1daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.732866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732849    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.732970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732877    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.733000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.732979    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.733031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733013    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp8sh\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.733066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733037    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.733066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733053    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.733127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733071    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.733216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733200    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.733253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733224    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.733286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733258    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.733316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.733298    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.833903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833816    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee37c72-f7bb-459c-9f06-df7401cc1daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.833903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833858    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.833903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833887    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833931    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833952    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp8sh\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.833982    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834005    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834036    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834097    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834123    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834180    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834252    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834297    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834324    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834356    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-kube-api-access-qqkzj\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834383    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834374    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834409    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834449    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.834544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834526    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.835095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834728    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.835095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834786    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.835095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834819    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.835095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834856    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.835095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.834919    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.835354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.835128    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aee37c72-f7bb-459c-9f06-df7401cc1daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.835354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.835301    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.836923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.836901    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.837202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.837178    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.837358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.837340    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.837423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.837401    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.843147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.843118    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.843273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.843180    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.844337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.844311    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqkzj\" (UniqueName: \"kubernetes.io/projected/aee37c72-f7bb-459c-9f06-df7401cc1daf-kube-api-access-qqkzj\") pod \"maas-default-gateway-openshift-default-769df7846c-b8m6d\" (UID: \"aee37c72-f7bb-459c-9f06-df7401cc1daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:56.844767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.844748    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp8sh\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\") pod \"maas-default-gateway-openshift-default-769df7846c-l2r8k\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.897995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.897956    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:56.922310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:56.922263    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:57.185697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.185664    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:24:57.187042 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:24:57.187008    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5df961_8bd7_46bb_8a90_1129a3840c56.slice/crio-65ddb4aaabc26e6582c60bd0b8a859afdd730c4288b31a8ee1b12773889d3337 WatchSource:0}: Error finding container 65ddb4aaabc26e6582c60bd0b8a859afdd730c4288b31a8ee1b12773889d3337: Status 404 returned error can't find the container with id 65ddb4aaabc26e6582c60bd0b8a859afdd730c4288b31a8ee1b12773889d3337
Oct 31 03:24:57.189344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.189305    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.189443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.189378    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.189443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.189405    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.216073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.216039    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"]
Oct 31 03:24:57.220501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.220469    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.220583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.220536    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.220583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.220565    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:24:57.595084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.595043    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" event={"ID":"aee37c72-f7bb-459c-9f06-df7401cc1daf","Type":"ContainerStarted","Data":"5cc2ffb659359277231cdf628e02121fd2cf1032d88e45e3876643cbff1bd0d2"}
Oct 31 03:24:57.595548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.595113    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" event={"ID":"aee37c72-f7bb-459c-9f06-df7401cc1daf","Type":"ContainerStarted","Data":"04f43f510474cdf2f3c3c19ec97fd53b78de972754481ca0a8510ac0fae8bea7"}
Oct 31 03:24:57.596754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.596727    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" event={"ID":"2d5df961-8bd7-46bb-8a90-1129a3840c56","Type":"ContainerStarted","Data":"2aafd46f3ceb4d244b016182aaaf742b49cda70ee0c5044a12b120f068682ce5"}
Oct 31 03:24:57.596899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.596769    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" event={"ID":"2d5df961-8bd7-46bb-8a90-1129a3840c56","Type":"ContainerStarted","Data":"65ddb4aaabc26e6582c60bd0b8a859afdd730c4288b31a8ee1b12773889d3337"}
Oct 31 03:24:57.650639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.648685    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podStartSLOduration=1.648665834 podStartE2EDuration="1.648665834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:57.647103146 +0000 UTC m=+572.172322734" watchObservedRunningTime="2026-10-31 03:24:57.648665834 +0000 UTC m=+572.173885421"
Oct 31 03:24:57.650639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.650465    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podStartSLOduration=1.650453593 podStartE2EDuration="1.650453593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:24:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:24:57.623936089 +0000 UTC m=+572.149155681" watchObservedRunningTime="2026-10-31 03:24:57.650453593 +0000 UTC m=+572.175673180"
Oct 31 03:24:57.898787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.898701    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:24:57.923349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:57.923318    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:24:58.900961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:58.900909    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:24:58.901490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:58.900980    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:24:58.924215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:58.924174    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:24:58.924384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:58.924232    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:24:59.899163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:59.899107    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:24:59.899386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:59.899184    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:24:59.923362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:59.923325    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:24:59.923794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:24:59.923380    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:00.900492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:00.900090    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:25:00.900492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:00.900162    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:25:00.923254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:00.923219    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:25:00.923394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:00.923270    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:01.107239 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:25:01.107116    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:25:01.899068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:01.899033    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:25:01.899317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:01.899289    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:25:01.923446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:01.923415    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:25:01.923900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:01.923704    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:02.475631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.474886    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"]
Oct 31 03:25:02.483709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.482885    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.489681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.489003    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-4rf7f\""
Oct 31 03:25:02.489681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.489061    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 03:25:02.489681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.489120    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 03:25:02.489949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.489716    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 03:25:02.493838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.493814    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 03:25:02.495097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.495072    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"]
Oct 31 03:25:02.530818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.530781    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j45b\" (UniqueName: \"kubernetes.io/projected/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-kube-api-access-5j45b\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.530987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.530956    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-metrics-certs\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.632438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.632399    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-metrics-certs\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.632656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.632451    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j45b\" (UniqueName: \"kubernetes.io/projected/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-kube-api-access-5j45b\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.635020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.634991    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-metrics-certs\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.641307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.641282    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j45b\" (UniqueName: \"kubernetes.io/projected/ceb67824-a366-4e5b-b4fa-4d8a48e898eb-kube-api-access-5j45b\") pod \"ai-gateway-controller-6fd9f75659-4s8nf\" (UID: \"ceb67824-a366-4e5b-b4fa-4d8a48e898eb\") " pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.822508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.822470    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:02.899692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.899106    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:25:02.899692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.899174    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:25:02.923360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.923326    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:25:02.923489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:02.923393    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:03.145113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.145042    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"]
Oct 31 03:25:03.634195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.634152    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf" event={"ID":"ceb67824-a366-4e5b-b4fa-4d8a48e898eb","Type":"ContainerStarted","Data":"f07970f079e531926b54bc1c0b9dd093839f6d41c694e6136c8cc632394ab750"}
Oct 31 03:25:03.898549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.898461    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:25:03.898549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.898522    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:25:03.923687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.923641    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:25:03.924369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:03.923709    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:04.899151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:04.899105    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body=
Oct 31 03:25:04.899322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:04.899175    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused"
Oct 31 03:25:04.923479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:04.923381    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Oct 31 03:25:04.923479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:04.923447    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Oct 31 03:25:06.222060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.222020    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:25:06.222926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.222676    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:25:06.223620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.223580    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:25:06.255970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.255937    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:25:06.256272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.256252    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:25:06.257179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.257148    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d"
Oct 31 03:25:06.657346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.657304    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf" event={"ID":"ceb67824-a366-4e5b-b4fa-4d8a48e898eb","Type":"ContainerStarted","Data":"8cff6e119ef2a2b9eddba50df18780b632df9c1dfc0da5f56ef76188ab5b6d6c"}
Oct 31 03:25:06.657561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.657376    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:06.676548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:06.676495    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf" podStartSLOduration=1.774722552 podStartE2EDuration="4.676480232s" totalImagesPullingTime="2.90175768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:06.674636402 +0000 UTC m=+581.199855984" watchObservedRunningTime="2026-10-31 03:25:06.676480232 +0000 UTC m=+581.201699818"
Oct 31 03:25:17.665693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:17.665664    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-6fd9f75659-4s8nf"
Oct 31 03:25:25.683670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:25.683638    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:25:25.687569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:25.687545    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:25:31.057742 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:25:31.057703    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:25:48.296394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.296360    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:25:48.305160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.305136    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.324739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.324711    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-090ebd-openshift-default-dockercfg-kj8g4\""
Oct 31 03:25:48.351685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.351646    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:25:48.388251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388210    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388252    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388270    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szppm\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388388    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388404    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388422    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388480    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388547    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.388780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.388573    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489425    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489467    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szppm\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489510    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489540    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489569    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489643    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489730    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.489833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.489764    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.490395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.490368    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.490542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.490446    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.490913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.490866    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.491008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.490992    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.491412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.491386    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.498488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.496918    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.498488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.497113    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.515225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.515186    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.517316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.517290    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szppm\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.552456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.552374    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:25:48.557586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.557564    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.570361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.570337    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f20dc4-openshift-default-dockercfg-zn4w2\""
Oct 31 03:25:48.590267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590227    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590275    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590365    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590416    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvkk\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590507    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590547    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590648    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590686    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.590776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.590739    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.596353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.596322    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:25:48.619254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.619217    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.691998    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692069    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692103    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692182    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692218    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692265    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvkk\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692330    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.692772    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.693020    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.693241    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.693766    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.694712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.694343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.704102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.704067    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.704738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.704717    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.717951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.717909    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.727639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.727267    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvkk\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.872471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.872384    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:48.944354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.944311    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:25:48.952963 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:48.952930    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c3a4316_41ab_453c_ac70_842d449fc475.slice/crio-a840234bf645f29e7691ecc1feb0f1d477a0292613001798cdd6dada3f382604 WatchSource:0}: Error finding container a840234bf645f29e7691ecc1feb0f1d477a0292613001798cdd6dada3f382604: Status 404 returned error can't find the container with id a840234bf645f29e7691ecc1feb0f1d477a0292613001798cdd6dada3f382604
Oct 31 03:25:48.956026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.955971    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:48.956132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.956070    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:48.956132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:48.956114    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.070753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.070720    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 03:25:49.075749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.075717    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095208    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095265    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095366    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095405    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095434    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095490    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095574    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095643    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vlh8\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.095788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.095687    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.123973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.123887    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-bcc7a4-openshift-default-dockercfg-pmpv4\""
Oct 31 03:25:49.136251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.136198    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 03:25:49.196214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196222    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196281    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196323    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196348    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196387    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196415    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196484    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vlh8\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196532    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196732    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.196860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.196847    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.197062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.197026    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.197183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.197073    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.197183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.197130    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.199474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.199447    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.199838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.199804    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.209757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.209730    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:25:49.212134 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:49.212099    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25c3543_bb26_44db_8eb0_7a0811d5a63e.slice/crio-c4b8f0dbbf669ce2ccd07f1978fd2f513a9a24f49477f8df7680f74551ab5d74 WatchSource:0}: Error finding container c4b8f0dbbf669ce2ccd07f1978fd2f513a9a24f49477f8df7680f74551ab5d74: Status 404 returned error can't find the container with id c4b8f0dbbf669ce2ccd07f1978fd2f513a9a24f49477f8df7680f74551ab5d74
Oct 31 03:25:49.212992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.212962    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.213238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.213205    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vlh8\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.215163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.215129    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.215368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.215339    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.215464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.215394    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.292818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.292787    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:25:49.297510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.297482    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.311021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.310994    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-dda5cb-openshift-default-dockercfg-z7cvq\""
Oct 31 03:25:49.392554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.392249    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:49.396772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.396255    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.397921    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398014    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398073    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9z4t\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398116    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398153    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398193    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398233    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398386    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.400918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.398477    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.500788    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.500872    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.500903    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.500931    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.500970    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9z4t\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.501009    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.501041    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.501071    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.501647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.501107    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.504014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.503479    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.504014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.503693    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.504014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.504010    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.504258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.504243    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.504481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.504458    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.505692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.504694    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.506510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.506390    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.515592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.515546    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.516643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.516592    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9z4t\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.616105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.616071    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:49.718094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.718059    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 03:25:49.718993 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:49.718964    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfa609e7_8ef4_41d3_a2da_3b7bf9dec882.slice/crio-336fc32b3cbdcdc18e6866c7ba14c325049c326ad702da9d1f9a6cf40712c41a WatchSource:0}: Error finding container 336fc32b3cbdcdc18e6866c7ba14c325049c326ad702da9d1f9a6cf40712c41a: Status 404 returned error can't find the container with id 336fc32b3cbdcdc18e6866c7ba14c325049c326ad702da9d1f9a6cf40712c41a
Oct 31 03:25:49.722061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.722027    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.722160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.722094    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.722160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.722124    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.894689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.894643    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerStarted","Data":"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"}
Oct 31 03:25:49.894898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.894717    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerStarted","Data":"336fc32b3cbdcdc18e6866c7ba14c325049c326ad702da9d1f9a6cf40712c41a"}
Oct 31 03:25:49.896544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.896512    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerStarted","Data":"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"}
Oct 31 03:25:49.896741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.896570    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerStarted","Data":"a840234bf645f29e7691ecc1feb0f1d477a0292613001798cdd6dada3f382604"}
Oct 31 03:25:49.898327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.898298    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerStarted","Data":"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"}
Oct 31 03:25:49.898461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.898356    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerStarted","Data":"c4b8f0dbbf669ce2ccd07f1978fd2f513a9a24f49477f8df7680f74551ab5d74"}
Oct 31 03:25:49.904469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.904443    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:25:49.905927 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:49.905903    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e137cd_412b_41c1_a5fa_fbbf9d47ef8d.slice/crio-3b4e3595a2c9949ba9fbe844f9dce319047b7c989e01af8345056637da2c9f44 WatchSource:0}: Error finding container 3b4e3595a2c9949ba9fbe844f9dce319047b7c989e01af8345056637da2c9f44: Status 404 returned error can't find the container with id 3b4e3595a2c9949ba9fbe844f9dce319047b7c989e01af8345056637da2c9f44
Oct 31 03:25:49.908831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.908801    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.908904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.908867    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.908904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.908898    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:25:49.940824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:49.940761    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podStartSLOduration=0.940741897 podStartE2EDuration="940.741897ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:49.939696655 +0000 UTC m=+624.464916244" watchObservedRunningTime="2026-10-31 03:25:49.940741897 +0000 UTC m=+624.465961486"
Oct 31 03:25:50.059839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.059775    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podStartSLOduration=2.059758452 podStartE2EDuration="2.059758452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:49.996099055 +0000 UTC m=+624.521318642" watchObservedRunningTime="2026-10-31 03:25:50.059758452 +0000 UTC m=+624.584978040"
Oct 31 03:25:50.393037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.392998    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:50.399382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.399354    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:50.453464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.453404    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podStartSLOduration=2.453388461 podStartE2EDuration="2.453388461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:50.059017009 +0000 UTC m=+624.584236600" watchObservedRunningTime="2026-10-31 03:25:50.453388461 +0000 UTC m=+624.978608050"
Oct 31 03:25:50.619744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.619696    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:50.625709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.625680    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:50.872888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.872852    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:50.878663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.878594    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:50.906418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.906379    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerStarted","Data":"72c7d4b7c99a49b73bb9a51a00de7412033631a1587e09eccb8fb97d8d5410fa"}
Oct 31 03:25:50.906736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.906701    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerStarted","Data":"3b4e3595a2c9949ba9fbe844f9dce319047b7c989e01af8345056637da2c9f44"}
Oct 31 03:25:50.907176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.907152    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:50.907543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.907520    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:50.908089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.908069    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:50.908188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.908146    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:25:50.908395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.908375    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:25:50.909063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.909043    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:25:50.968484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:50.968421    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podStartSLOduration=1.9684060570000002 podStartE2EDuration="1.968406057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:25:50.93906133 +0000 UTC m=+625.464280937" watchObservedRunningTime="2026-10-31 03:25:50.968406057 +0000 UTC m=+625.493625648"
Oct 31 03:25:51.617012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:51.616978    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:51.623183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:51.623151    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:51.912706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:51.912588    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:51.913639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:51.913601    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:25:55.579160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.579124    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:25:55.584095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.584070    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.621947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.621915    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:25:55.634743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.634713    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-090ebd\""
Oct 31 03:25:55.634909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.634812    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-090ebd-dockercfg-cqqqg\""
Oct 31 03:25:55.698904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.698866    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfpc6\" (UniqueName: \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.699096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.698930    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.801130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.801085    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfpc6\" (UniqueName: \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.801332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.801191    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.801958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.801938    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.824072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.824033    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfpc6\" (UniqueName: \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\") pod \"payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:55.897717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:55.897632    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:25:56.247362 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:56.247325    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25971da9_a3af_42ec_b57a_c36000f39d0e.slice/crio-f62d2f4ff572b3f4c470370260632c84edfa8861d29e4b8a9710d38061875bec WatchSource:0}: Error finding container f62d2f4ff572b3f4c470370260632c84edfa8861d29e4b8a9710d38061875bec: Status 404 returned error can't find the container with id f62d2f4ff572b3f4c470370260632c84edfa8861d29e4b8a9710d38061875bec
Oct 31 03:25:56.249667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:56.249636    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:25:56.250181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:56.250164    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:25:56.943114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:56.943074    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" event={"ID":"25971da9-a3af-42ec-b57a-c36000f39d0e","Type":"ContainerStarted","Data":"f62d2f4ff572b3f4c470370260632c84edfa8861d29e4b8a9710d38061875bec"}
Oct 31 03:25:57.218650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.218549    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:25:57.227644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.222911    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.247022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.246982    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:25:57.319634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.318885    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f20dc4\""
Oct 31 03:25:57.319634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.318978    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f20dc4-dockercfg-wdx5k\""
Oct 31 03:25:57.319634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.319580    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.319954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.319764    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlxms\" (UniqueName: \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.420697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.420645    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlxms\" (UniqueName: \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.420897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.420785    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.431903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.431864    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.449550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.449323    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlxms\" (UniqueName: \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\") pod \"payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.541082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.541048    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:57.843068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.843032    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:25:57.843965 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:57.843936    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48accdc9_4ac0_43fe_b087_863bcbb69aa2.slice/crio-017251d5e822aeb2be89931f58d912a215d7715a437553b3dc2cbb9c00cd2e9d WatchSource:0}: Error finding container 017251d5e822aeb2be89931f58d912a215d7715a437553b3dc2cbb9c00cd2e9d: Status 404 returned error can't find the container with id 017251d5e822aeb2be89931f58d912a215d7715a437553b3dc2cbb9c00cd2e9d
Oct 31 03:25:57.950161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:57.950128    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" event={"ID":"48accdc9-4ac0-43fe-b087-863bcbb69aa2","Type":"ContainerStarted","Data":"017251d5e822aeb2be89931f58d912a215d7715a437553b3dc2cbb9c00cd2e9d"}
Oct 31 03:25:58.348816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.348774    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 03:25:58.354812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.354780    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.370876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.370841    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 03:25:58.444637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.444274    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-bcc7a4-dockercfg-mzz4z\""
Oct 31 03:25:58.444637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.444419    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-bcc7a4\""
Oct 31 03:25:58.446297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.446260    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.446450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.446405    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.549496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.548013    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.549496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.548168    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.549496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.549426    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.564891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.564812    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\") pod \"payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.678954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.678855    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:58.961857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.961779    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 03:25:58.967247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.967219    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:58.982711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.982681    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-7f3159\""
Oct 31 03:25:58.982854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.982801    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-7f3159-dockercfg-25kcg\""
Oct 31 03:25:59.000773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:58.999811    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 03:25:59.009439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.009404    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 03:25:59.009587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.009543    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.011293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.011269    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 03:25:59.029409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.029377    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 03:25:59.054337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.054215    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk5dr\" (UniqueName: \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.054790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.054739    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.157713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.157064    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.157713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.157201    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hknrn\" (UniqueName: \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.157713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.157325    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk5dr\" (UniqueName: \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.157713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.157495    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.158508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.158200    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.186876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.186803    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk5dr\" (UniqueName: \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\") pod \"payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.259896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.259790    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.260075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.259928    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hknrn\" (UniqueName: \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.260824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.260799    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.271197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.271165    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hknrn\" (UniqueName: \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\") pod \"payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.286686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.286364    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:25:59.345037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.344812    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:25:59.617736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.617660    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 03:25:59.690338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.690302    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 03:25:59.695343 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:25:59.695312    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc210d02_f4a2_4f76_8a6f_12484b05d3ac.slice/crio-0cefbe77bf926fb2c19f7bcb5ca5fc2c9eff6a84498906680b994a0bcc6e0a37 WatchSource:0}: Error finding container 0cefbe77bf926fb2c19f7bcb5ca5fc2c9eff6a84498906680b994a0bcc6e0a37: Status 404 returned error can't find the container with id 0cefbe77bf926fb2c19f7bcb5ca5fc2c9eff6a84498906680b994a0bcc6e0a37
Oct 31 03:25:59.966722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.966670    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" event={"ID":"48accdc9-4ac0-43fe-b087-863bcbb69aa2","Type":"ContainerStarted","Data":"e3b0eeeacdf8fd1beb635276f72434db3b90a77204335b39db42e38d73343843"}
Oct 31 03:25:59.967271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.966901    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:25:59.969670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.969643    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" event={"ID":"fc210d02-f4a2-4f76-8a6f-12484b05d3ac","Type":"ContainerStarted","Data":"0cefbe77bf926fb2c19f7bcb5ca5fc2c9eff6a84498906680b994a0bcc6e0a37"}
Oct 31 03:25:59.970991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.970966    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" event={"ID":"cab7e371-cb9c-46ca-be9c-faa8000ba33f","Type":"ContainerStarted","Data":"4ad73de03959d53fb8a3f7cc08fcef76426a9a8235b89f51ad05c00757b9e5b2"}
Oct 31 03:25:59.972522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.972488    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" event={"ID":"8a4810ca-5db9-4232-a5fe-1abf2b78cc54","Type":"ContainerStarted","Data":"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"}
Oct 31 03:25:59.972615 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.972527    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" event={"ID":"8a4810ca-5db9-4232-a5fe-1abf2b78cc54","Type":"ContainerStarted","Data":"8622b93fb7df721c23584ad4805decafb76cc284f0b33c0182187ba54f6b6c09"}
Oct 31 03:25:59.972615 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.972563    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:25:59.974103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.974080    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" event={"ID":"25971da9-a3af-42ec-b57a-c36000f39d0e","Type":"ContainerStarted","Data":"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"}
Oct 31 03:25:59.974255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:25:59.974239    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:26:00.002839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.002796    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" podStartSLOduration=1.069639026 podStartE2EDuration="3.002780856s" totalImagesPullingTime="1.93314183s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:00.000997487 +0000 UTC m=+634.526217076" watchObservedRunningTime="2026-10-31 03:26:00.002780856 +0000 UTC m=+634.528000444"
Oct 31 03:26:00.039276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.039214    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podStartSLOduration=1.465112542 podStartE2EDuration="5.039196727s" totalImagesPullingTime="3.574084185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:00.038094988 +0000 UTC m=+634.563314576" watchObservedRunningTime="2026-10-31 03:26:00.039196727 +0000 UTC m=+634.564416316"
Oct 31 03:26:00.089601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.089546    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" podStartSLOduration=1.3172433940000001 podStartE2EDuration="2.089526393s" totalImagesPullingTime="772.282999ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:00.085960633 +0000 UTC m=+634.611180216" watchObservedRunningTime="2026-10-31 03:26:00.089526393 +0000 UTC m=+634.614745981"
Oct 31 03:26:00.306250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.306214    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:26:00.381912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.381832    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:26:00.385855 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.385832    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.408588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.408551    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:26:00.408819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.408802    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-6c6050\""
Oct 31 03:26:00.474921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.474883    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.475116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.474940    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hls5k\" (UniqueName: \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.585631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.585543    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.595638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.591661    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.595638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.591781    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hls5k\" (UniqueName: \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.612003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.611582    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hls5k\" (UniqueName: \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\") pod \"payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.708567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.705789    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:00.983143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.983041    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" event={"ID":"fc210d02-f4a2-4f76-8a6f-12484b05d3ac","Type":"ContainerStarted","Data":"e41ddcfdb9b9aa77620d0a2354742f69060fc252a358b725cd3d34263ac3a457"}
Oct 31 03:26:00.983583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.983161    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:26:00.985076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:00.985037    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" event={"ID":"cab7e371-cb9c-46ca-be9c-faa8000ba33f","Type":"ContainerStarted","Data":"d82b37c183cdc2decb3f215c26b787f56f70525674b28eab02a0326817c16eca"}
Oct 31 03:26:01.002750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.002695    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" podStartSLOduration=2.585676428 podStartE2EDuration="3.002679116s" totalImagesPullingTime="417.002688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:01.000382072 +0000 UTC m=+635.525601688" watchObservedRunningTime="2026-10-31 03:26:01.002679116 +0000 UTC m=+635.527898723"
Oct 31 03:26:01.027310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.027280    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:26:01.028937 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:26:01.028902    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0374ab_5f65_4545_ba95_bda2f7a56913.slice/crio-95f524d1d740c2fc4d3438c0098a2611a8f9a8866666d16bc967cbe58a6acb68 WatchSource:0}: Error finding container 95f524d1d740c2fc4d3438c0098a2611a8f9a8866666d16bc967cbe58a6acb68: Status 404 returned error can't find the container with id 95f524d1d740c2fc4d3438c0098a2611a8f9a8866666d16bc967cbe58a6acb68
Oct 31 03:26:01.032466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.032413    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" podStartSLOduration=2.659115092 podStartE2EDuration="3.032389261s" totalImagesPullingTime="373.274169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:25:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:01.0287021 +0000 UTC m=+635.553921689" watchObservedRunningTime="2026-10-31 03:26:01.032389261 +0000 UTC m=+635.557608851"
Oct 31 03:26:01.059825 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:26:01.059775    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:26:01.073327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.073299    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:26:01.078832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.078808    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.086105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.086063    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-dda5cb-dockercfg-zxdj9\""
Oct 31 03:26:01.089421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.089388    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:26:01.166354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.166322    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-dda5cb\""
Oct 31 03:26:01.209624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.209236    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.209624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.209372    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgk42\" (UniqueName: \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.310316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.310274    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgk42\" (UniqueName: \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.310538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.310452    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.311423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.311397    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.321302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.321267    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgk42\" (UniqueName: \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\") pod \"payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.401036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.399731    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.775693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.775576    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:26:01.782787 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:26:01.782746    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03172c7_d030_405c_a795_5313a22c806d.slice/crio-66268a9b7a41ba440f86bf5e8b841b5461ca84937ecb951873849d60631dbb7a WatchSource:0}: Error finding container 66268a9b7a41ba440f86bf5e8b841b5461ca84937ecb951873849d60631dbb7a: Status 404 returned error can't find the container with id 66268a9b7a41ba440f86bf5e8b841b5461ca84937ecb951873849d60631dbb7a
Oct 31 03:26:01.994576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.994532    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" event={"ID":"d03172c7-d030-405c-a795-5313a22c806d","Type":"ContainerStarted","Data":"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"}
Oct 31 03:26:01.995077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.994625    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" event={"ID":"d03172c7-d030-405c-a795-5313a22c806d","Type":"ContainerStarted","Data":"66268a9b7a41ba440f86bf5e8b841b5461ca84937ecb951873849d60631dbb7a"}
Oct 31 03:26:01.995077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.994656    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:01.996272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.996241    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" event={"ID":"9a0374ab-5f65-4545-ba95-bda2f7a56913","Type":"ContainerStarted","Data":"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"}
Oct 31 03:26:01.996384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.996297    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" event={"ID":"9a0374ab-5f65-4545-ba95-bda2f7a56913","Type":"ContainerStarted","Data":"95f524d1d740c2fc4d3438c0098a2611a8f9a8866666d16bc967cbe58a6acb68"}
Oct 31 03:26:01.996771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:01.996747    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:26:02.013262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:02.013191    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" podStartSLOduration=1.013170901 podStartE2EDuration="1.013170901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:02.012224003 +0000 UTC m=+636.537443595" watchObservedRunningTime="2026-10-31 03:26:02.013170901 +0000 UTC m=+636.538390490"
Oct 31 03:26:03.005586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:03.004950    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:09.381349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.381278    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" podStartSLOduration=9.381253932 podStartE2EDuration="9.381253932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:02.03480223 +0000 UTC m=+636.560021839" watchObservedRunningTime="2026-10-31 03:26:09.381253932 +0000 UTC m=+643.906473522"
Oct 31 03:26:09.383240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.383192    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:26:09.418833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.418797    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:26:09.419196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.419172    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.441862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.434987    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"simulator-w0-d86a77-kserve-self-signed-certs\""
Oct 31 03:26:09.441862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.435129    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"default-dockercfg-qfn76\""
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442713    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442797    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442832    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442882    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442959    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.443962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.442991    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfn6q\" (UniqueName: \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.537627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.523321    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"openshift-service-ca.crt\""
Oct 31 03:26:09.537627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.523442    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d86a77\"/\"kube-root-ca.crt\""
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544639    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544700    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544720    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544753    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544800    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.544824    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfn6q\" (UniqueName: \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.545196    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.545806    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.545826    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.548480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.548454    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.548651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.548578    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.742728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.726747    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:26:09.747824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.747783    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfn6q\" (UniqueName: \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\") pod \"simulator-w0-d86a77-kserve-5b788df88b-rfplw\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") " pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:09.930412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:09.930066    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:10.630908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:10.630838    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:26:11.028584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.026820    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:26:11.028584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.027831    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 03:26:11.039653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.039020    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:26:11.246187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.246091    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" event={"ID":"70ba18cb-cbc9-46e0-af51-873aba870133","Type":"ContainerStarted","Data":"0ac9d4f2852554fc58ccbff26ace61753036f373c77387e92fd90388d7b1899a"}
Oct 31 03:26:11.246187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.246158    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" event={"ID":"70ba18cb-cbc9-46e0-af51-873aba870133","Type":"ContainerStarted","Data":"95f7456853352e9dc8b5fcc90708907e10308445040d9a2463acd03f311f470c"}
Oct 31 03:26:11.293208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.282569    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podStartSLOduration=2.282543444 podStartE2EDuration="2.282543444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:11.269116557 +0000 UTC m=+645.794336174" watchObservedRunningTime="2026-10-31 03:26:11.282543444 +0000 UTC m=+645.807763037"
Oct 31 03:26:11.938312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:11.938276    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:12.004497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:12.004458    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 03:26:13.007667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:13.007531    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 03:26:13.008640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:13.008583    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:26:14.016947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:14.015825    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:26:18.777672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:18.777638    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:26:19.322259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.322213    5098 generic.go:356] "Generic (PLEG): container finished" podID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerID="347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444" exitCode=0
Oct 31 03:26:19.322463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.322286    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerDied","Data":"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"}
Oct 31 03:26:19.323270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.323246    5098 scope.go:122] "RemoveContainer" containerID="347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"
Oct 31 03:26:19.324768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.324744    5098 generic.go:356] "Generic (PLEG): container finished" podID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerID="e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740" exitCode=0
Oct 31 03:26:19.324891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.324819    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerDied","Data":"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"}
Oct 31 03:26:19.324891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.324824    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.325018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.324910    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.325018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.324954    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.325218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.325200    5098 scope.go:122] "RemoveContainer" containerID="e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"
Oct 31 03:26:19.326686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.326656    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.326764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.326738    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.326807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.326787    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:19.937805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.937756    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:19.981705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:19.977988    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:20.332758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.332723    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerStarted","Data":"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"}
Oct 31 03:26:20.334849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.334825    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerStarted","Data":"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"}
Oct 31 03:26:20.336078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.336055    5098 generic.go:356] "Generic (PLEG): container finished" podID="1c3a4316-41ab-453c-ac70-842d449fc475" containerID="68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361" exitCode=0
Oct 31 03:26:20.336128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.336116    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerDied","Data":"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"}
Oct 31 03:26:20.336414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.336400    5098 scope.go:122] "RemoveContainer" containerID="68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"
Oct 31 03:26:20.337519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.337493    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.337574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.337562    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.337622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.337595    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.338988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.338967    5098 generic.go:356] "Generic (PLEG): container finished" podID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerID="72c7d4b7c99a49b73bb9a51a00de7412033631a1587e09eccb8fb97d8d5410fa" exitCode=0
Oct 31 03:26:20.339113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.339096    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerDied","Data":"72c7d4b7c99a49b73bb9a51a00de7412033631a1587e09eccb8fb97d8d5410fa"}
Oct 31 03:26:20.339602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.339554    5098 scope.go:122] "RemoveContainer" containerID="72c7d4b7c99a49b73bb9a51a00de7412033631a1587e09eccb8fb97d8d5410fa"
Oct 31 03:26:20.340876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.340850    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.340929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.340920    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.340967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.340952    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:20.400660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.394008    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:26:20.400660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.394159    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:26:20.400660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.394258    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:20.400660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.394297    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:20.621431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.619422    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:26:20.631513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.631482    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.668655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.644356    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:26:20.668655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.646347    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"default-dockercfg-fnwtp\""
Oct 31 03:26:20.668655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.646415    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"simulator-w3-090ebd-kserve-self-signed-certs\""
Oct 31 03:26:20.668655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.647142    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"openshift-service-ca.crt\""
Oct 31 03:26:20.668655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.647201    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-090ebd\"/\"kube-root-ca.crt\""
Oct 31 03:26:20.742691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.742486    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.742691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.742537    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.742691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.742627    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.742691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.742668    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.746695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.745945    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wj9\" (UniqueName: \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.746695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.746038    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.746695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.746253    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:26:20.823652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.811014    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.837521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.832863    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:26:20.847008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.846975    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 03:26:20.847462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847421    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.847619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847585    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.847720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847665    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.847720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847703    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.847848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847730    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.847848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847791    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-hwjnd\""
Oct 31 03:26:20.847951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847850    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.847951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847877    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 03:26:20.847951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.848094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.847999    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wj9\" (UniqueName: \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.848094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.848062    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.848197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.848096    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.848197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.848121    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnvbv\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.848197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.848178    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.848343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.848213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.869213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.851837    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.869213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.852094    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.869213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.852374    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.869213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.867966    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.902400    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.927817    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.928515    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.928645    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.928702    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.928961    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.931637    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:20.931711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.931697    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:20.933704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.933680    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wj9\" (UniqueName: \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\") pod \"simulator-w3-090ebd-kserve-b559b5748-22hs5\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") " pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956418    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956478    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnvbv\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956531    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956566    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956641    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956693    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.957629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.956727    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:20.981490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:20.981452    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.027429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.006087    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.043851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.043721    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.045245 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.044899    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.068868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.048627    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnvbv\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.068868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.060099    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.083629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.083563    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-zl6d8\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.088643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.086445    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:21.182682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.180866    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:21.406928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.406278    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:21.406928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.406340    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:21.418920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.417936    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerStarted","Data":"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"}
Oct 31 03:26:21.425848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.425464    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerStarted","Data":"03debde827ac5a10e924272ea14f621281a735293ddd56c958d5ec3f19d94115"}
Oct 31 03:26:21.535116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.524286    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:26:21.616602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.616561    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:26:21.617357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.617330    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:21.617464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.617385    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:21.619998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.619973    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:26:21.620669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.620648    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:21.620770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.620695    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:21.641195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.641171    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:26:21.642530 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:26:21.642499    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb483b06_2e94_4645_8bdd_3a666a6c4b72.slice/crio-aab7dad118e0c526a7a4fbdf77f867caa4d9c1c84dc6887b80cdfb50e661a5e8 WatchSource:0}: Error finding container aab7dad118e0c526a7a4fbdf77f867caa4d9c1c84dc6887b80cdfb50e661a5e8: Status 404 returned error can't find the container with id aab7dad118e0c526a7a4fbdf77f867caa4d9c1c84dc6887b80cdfb50e661a5e8
Oct 31 03:26:21.873094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.873024    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:21.873762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.873698    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:21.918843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:21.918745    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:26:22.394632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.393786    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:22.394632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.393852    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:22.515099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.512731    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" event={"ID":"de93c07b-3571-4b99-a12f-8e40941f28c2","Type":"ContainerStarted","Data":"1b69b32eecc83054dff71a1a056eda3e4cc0110526f039ecbdad8ad104ee74e1"}
Oct 31 03:26:22.519644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.518187    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" event={"ID":"bb483b06-2e94-4645-8bdd-3a666a6c4b72","Type":"ContainerStarted","Data":"ec755e445033e26b390973b28862803752959e0d45a4069f5694331978bd55ad"}
Oct 31 03:26:22.519644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.518233    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" event={"ID":"bb483b06-2e94-4645-8bdd-3a666a6c4b72","Type":"ContainerStarted","Data":"aab7dad118e0c526a7a4fbdf77f867caa4d9c1c84dc6887b80cdfb50e661a5e8"}
Oct 31 03:26:22.620088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.620051    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:22.620688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.620658    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:22.621218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.620568    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:22.628237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.628201    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:22.876484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.876017    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:22.876484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:22.876083    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:23.199937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.199794    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:23.395140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.395106    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:23.395776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.395730    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:23.622634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.621047    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:23.622634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.621111    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:23.622634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.621764    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:23.622634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.621802    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:23.885191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.885098    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:23.885455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:23.885426    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:24.398513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.398045    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:24.398513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.398112    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:24.464265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.464200    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:26:24.630273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.630237    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:24.630590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.630544    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:24.631560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.631520    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:24.631748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.631725    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:24.881634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.877167    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:24.881634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:24.877230    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:25.351633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.342712    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:26:25.390633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.378696    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:26:25.396117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.396079    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:25.396354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.396323    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:25.477652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.477592    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:26:25.480601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.479789    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:26:25.513665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.510110    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podStartSLOduration=5.510088609 podStartE2EDuration="5.510088609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:22.546573483 +0000 UTC m=+657.071793072" watchObservedRunningTime="2026-10-31 03:26:25.510088609 +0000 UTC m=+660.035308199"
Oct 31 03:26:25.700830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.700727    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:25.700830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.700790    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:25.706632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.703204    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:25.706632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.703258    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:25.918126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.918087    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Oct 31 03:26:25.918450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:25.918412    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Oct 31 03:26:26.121078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.121039    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:26:26.319766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.319721    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:26:26.431797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.431705    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Oct 31 03:26:26.432045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.432011    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Oct 31 03:26:26.618563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.618527    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:26.619160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.619128    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:26.621261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.621237    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:26.621427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.621407    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:26.913035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.913005    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:26:26.914916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:26.914891    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:26:27.473920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.472417    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:26:27.501642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.487955    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 03:26:27.619019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.618981    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 03:26:27.619412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.619043    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 03:26:27.619571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.619549    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Oct 31 03:26:27.619633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:27.619588    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Oct 31 03:26:28.820849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:28.819923    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:26:28.821643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:28.821562    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:26:28.901689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:28.900348    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:26:28.902017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:28.901975    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:26:28.902757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:28.902732    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:26:29.855791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:29.855752    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:29.856192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:29.855840    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:26:30.328633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:30.328567    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" event={"ID":"de93c07b-3571-4b99-a12f-8e40941f28c2","Type":"ContainerStarted","Data":"f2f279491312e817c79328be6d5ed1a8166f9cde71c03caf09094b2b05adb4bd"}
Oct 31 03:26:30.329752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:30.329699    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:30.331298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:30.331016    5098 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zl6d8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:26:30.331298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:30.331069    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:26:30.352677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:30.352581    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" podStartSLOduration=2.037535047 podStartE2EDuration="10.352567162s" totalImagesPullingTime="8.315032115s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:26:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:26:30.348308217 +0000 UTC m=+664.873527817" watchObservedRunningTime="2026-10-31 03:26:30.352567162 +0000 UTC m=+664.877786751"
Oct 31 03:26:31.058818 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:26:31.058781    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:26:31.188284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:31.188250    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:31.201323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:31.201295    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:31.334666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:31.334562    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:26:31.347471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:31.347439    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:26:36.210224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:36.210136    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:26:52.183405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:26:52.183368    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:27:01.058141 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:27:01.058095    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:27:04.053886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.053844    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:27:04.063167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.063140    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.068803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.068767    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-6c6050-openshift-default-dockercfg-wsmcd\""
Oct 31 03:27:04.069337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.069307    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:27:04.095544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095499    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.095783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095637    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx2n6\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.095783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095765    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.095924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095801    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.095924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095824    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.095924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095844    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.096085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095960    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.096085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.095993    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.096085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.096028    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.196883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.196834    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.196883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.196883    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.196904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.196925    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.196991    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197028    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197074    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197149    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197177    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx2n6\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197371    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197500    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197637    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197745    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.197812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.197765    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.199494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.199477    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.199768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.199751    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.210831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.210802    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.210965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.210872    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx2n6\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.384307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.384224    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:04.671135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.671097    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:27:04.672402 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:04.672368    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78defb67_9daa_4f22_9580_02319bcae539.slice/crio-1fb01762274befd28b1ae5db23fd09489732abecb39b3e7b99211f0e6aba68c2 WatchSource:0}: Error finding container 1fb01762274befd28b1ae5db23fd09489732abecb39b3e7b99211f0e6aba68c2: Status 404 returned error can't find the container with id 1fb01762274befd28b1ae5db23fd09489732abecb39b3e7b99211f0e6aba68c2
Oct 31 03:27:04.675041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.674998    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:04.675144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.675092    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:04.675184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:04.675140    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:27:05.580487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:05.580447    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" event={"ID":"78defb67-9daa-4f22-9580-02319bcae539","Type":"ContainerStarted","Data":"280bb8eda60e469b904b1a86f13b7575c5c62d9b56d41083dd4c13f4f0ec1162"}
Oct 31 03:27:05.580990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:05.580513    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" event={"ID":"78defb67-9daa-4f22-9580-02319bcae539","Type":"ContainerStarted","Data":"1fb01762274befd28b1ae5db23fd09489732abecb39b3e7b99211f0e6aba68c2"}
Oct 31 03:27:05.608908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:05.608857    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podStartSLOduration=1.608842305 podStartE2EDuration="1.608842305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:05.605015938 +0000 UTC m=+700.130235526" watchObservedRunningTime="2026-10-31 03:27:05.608842305 +0000 UTC m=+700.134061893"
Oct 31 03:27:06.384902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:06.384859    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:06.386689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:06.386664    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 03:27:06.386820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:06.386712    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 03:27:07.388632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:07.388282    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 03:27:07.388632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:07.388404    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 03:27:08.388131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:08.388090    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 03:27:08.388476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:08.388440    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 03:27:08.447562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:08.447525    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:27:09.385016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:09.384973    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 03:27:09.385197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:09.385058    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 03:27:10.385304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:10.385260    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 03:27:10.385766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:10.385325    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 03:27:11.875747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:11.875713    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:11.876289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:11.876269    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:11.877457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:11.877433    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:27:29.774967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:29.774933    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:27:31.079554 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:27:31.079510    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:27:41.192778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.192739    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"]
Oct 31 03:27:41.197528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.197503    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.201478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.201442    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:41.201647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.201507    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:41.202864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.202840    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:41.202984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.202892    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:27:41.204086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.204058    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"]
Oct 31 03:27:41.286969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.286930    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.287149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.286992    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.287149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.287101    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jts2g\" (UniqueName: \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.388422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.388380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.388643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.388452    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.388643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.388491    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jts2g\" (UniqueName: \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.389236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.389211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.390943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.390922    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.398884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.398854    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jts2g\" (UniqueName: \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\") pod \"maas-api-revoke-sub-6439353e-xxjhh\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.515223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.515145    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:41.795877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.795846    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"]
Oct 31 03:27:41.796866 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:41.796830    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8254ae_c7d1_4d3c_ac62_b2aa90b2ff9c.slice/crio-d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16 WatchSource:0}: Error finding container d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16: Status 404 returned error can't find the container with id d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16
Oct 31 03:27:41.815463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.815434    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh" event={"ID":"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c","Type":"ContainerStarted","Data":"d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16"}
Oct 31 03:27:41.901511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.901472    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"]
Oct 31 03:27:41.906789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.906766    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:41.912425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.912398    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"]
Oct 31 03:27:41.995497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.995459    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvxp4\" (UniqueName: \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:41.995749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.995515    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:41.995749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:41.995727    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.096978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.096881    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvxp4\" (UniqueName: \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.096978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.096973    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.097210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.097061    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.097827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.097799    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.099438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.099416    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.105116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.105091    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvxp4\" (UniqueName: \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\") pod \"maas-api-revoke-sub-81a649c6-86qbq\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.224595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.224556    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:42.503155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.503120    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"]
Oct 31 03:27:42.503836 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:42.503811    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2e0c5e6_691c_4896_af67_30415a668321.slice/crio-979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709 WatchSource:0}: Error finding container 979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709: Status 404 returned error can't find the container with id 979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709
Oct 31 03:27:42.823251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:42.823216    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq" event={"ID":"c2e0c5e6-691c-4896-af67-30415a668321","Type":"ContainerStarted","Data":"979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709"}
Oct 31 03:27:45.932626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.932580    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"]
Oct 31 03:27:45.938171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.938145    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:45.943487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.943460    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"]
Oct 31 03:27:45.949477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.949445    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:45.949634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.949495    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:45.949708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:45.949589    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6rdk\" (UniqueName: \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.050968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.050929    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.050968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.050967    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.051215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.050985    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6rdk\" (UniqueName: \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.051823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.051787    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.053753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.053730    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.060309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.060284    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6rdk\" (UniqueName: \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\") pod \"maas-api-revoke-sub-dac23050-t6thb\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.271036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.271011    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:46.563456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.563425    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"]
Oct 31 03:27:46.563773 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:46.563748    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d51d22_8ae5_47f6_bd4c_b165017f91b4.slice/crio-e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928 WatchSource:0}: Error finding container e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928: Status 404 returned error can't find the container with id e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928
Oct 31 03:27:46.857082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.857046    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb" event={"ID":"a8d51d22-8ae5-47f6-bd4c-b165017f91b4","Type":"ContainerStarted","Data":"e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928"}
Oct 31 03:27:46.858863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.858835    5098 generic.go:356] "Generic (PLEG): container finished" podID="4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" containerID="5c996a8d3341b9c901156db828b4dc74fa22175bd97cd99e46b4ec74f59c889b" exitCode=0
Oct 31 03:27:46.858970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.858925    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh" event={"ID":"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c","Type":"ContainerDied","Data":"5c996a8d3341b9c901156db828b4dc74fa22175bd97cd99e46b4ec74f59c889b"}
Oct 31 03:27:46.861026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.860996    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq" event={"ID":"c2e0c5e6-691c-4896-af67-30415a668321","Type":"ContainerStarted","Data":"547b257b696d03b6b200b7c4bc70a950cdf7e09277b4e15c7736bf7186fc5a0e"}
Oct 31 03:27:46.898434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:46.898368    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq" podStartSLOduration=2.494638731 podStartE2EDuration="5.898344886s" totalImagesPullingTime="3.403706155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:27:46.896414205 +0000 UTC m=+741.421633795" watchObservedRunningTime="2026-10-31 03:27:46.898344886 +0000 UTC m=+741.423564475"
Oct 31 03:27:47.878462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:47.878424    5098 generic.go:356] "Generic (PLEG): container finished" podID="a8d51d22-8ae5-47f6-bd4c-b165017f91b4" containerID="42a3e3a384b74a957251b0b99e4a0ca8cd5a187da6a1b08874e2b792d3e6dfc8" exitCode=0
Oct 31 03:27:47.878947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:47.878513    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb" event={"ID":"a8d51d22-8ae5-47f6-bd4c-b165017f91b4","Type":"ContainerDied","Data":"42a3e3a384b74a957251b0b99e4a0ca8cd5a187da6a1b08874e2b792d3e6dfc8"}
Oct 31 03:27:47.880385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:47.880356    5098 generic.go:356] "Generic (PLEG): container finished" podID="c2e0c5e6-691c-4896-af67-30415a668321" containerID="547b257b696d03b6b200b7c4bc70a950cdf7e09277b4e15c7736bf7186fc5a0e" exitCode=0
Oct 31 03:27:47.880506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:47.880437    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq" event={"ID":"c2e0c5e6-691c-4896-af67-30415a668321","Type":"ContainerDied","Data":"547b257b696d03b6b200b7c4bc70a950cdf7e09277b4e15c7736bf7186fc5a0e"}
Oct 31 03:27:48.042300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.042273    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:48.076319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.076286    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\") pod \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") "
Oct 31 03:27:48.076319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.076322    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\" (UniqueName: \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\") pod \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") "
Oct 31 03:27:48.076753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.076723    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca" pod "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" (UID: "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:48.078659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.078630    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g" pod "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" (UID: "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c"). InnerVolumeSpecName "kube-api-access-jts2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:48.177051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.176954    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\") pod \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\" (UID: \"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c\") "
Oct 31 03:27:48.177394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.177378    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:48.177439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.177397    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jts2g\" (UniqueName: \"kubernetes.io/projected/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-kube-api-access-jts2g\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:48.178996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.178970    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca" pod "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" (UID: "4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:48.279602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.279551    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:48.889021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.888988    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"
Oct 31 03:27:48.889021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.889013    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh" event={"ID":"4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c","Type":"ContainerDied","Data":"d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16"}
Oct 31 03:27:48.889530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:48.889065    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d79710763e7ab0c51fb86240aee23a55b01e14ff87835fa496d9f4e716cdba16"
Oct 31 03:27:49.091641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.091591    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:49.095641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.095617    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:49.192181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192095    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\" (UniqueName: \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\") pod \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") "
Oct 31 03:27:49.192181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192163    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\" (UniqueName: \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\") pod \"c2e0c5e6-691c-4896-af67-30415a668321\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") "
Oct 31 03:27:49.192365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192222    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\") pod \"c2e0c5e6-691c-4896-af67-30415a668321\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") "
Oct 31 03:27:49.192365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192266    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\") pod \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") "
Oct 31 03:27:49.192457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192436    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\") pod \"c2e0c5e6-691c-4896-af67-30415a668321\" (UID: \"c2e0c5e6-691c-4896-af67-30415a668321\") "
Oct 31 03:27:49.192511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\") pod \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\" (UID: \"a8d51d22-8ae5-47f6-bd4c-b165017f91b4\") "
Oct 31 03:27:49.192669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192632    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca" pod "c2e0c5e6-691c-4896-af67-30415a668321" (UID: "c2e0c5e6-691c-4896-af67-30415a668321"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:49.192811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192753    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca" pod "a8d51d22-8ae5-47f6-bd4c-b165017f91b4" (UID: "a8d51d22-8ae5-47f6-bd4c-b165017f91b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:49.193019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.192994    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.193125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.193022    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.194495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.194463    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca" pod "c2e0c5e6-691c-4896-af67-30415a668321" (UID: "c2e0c5e6-691c-4896-af67-30415a668321"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:49.194599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.194495    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk" pod "a8d51d22-8ae5-47f6-bd4c-b165017f91b4" (UID: "a8d51d22-8ae5-47f6-bd4c-b165017f91b4"). InnerVolumeSpecName "kube-api-access-j6rdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:49.194599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.194548    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca" pod "a8d51d22-8ae5-47f6-bd4c-b165017f91b4" (UID: "a8d51d22-8ae5-47f6-bd4c-b165017f91b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:49.194599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.194624    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4" pod "c2e0c5e6-691c-4896-af67-30415a668321" (UID: "c2e0c5e6-691c-4896-af67-30415a668321"). InnerVolumeSpecName "kube-api-access-vvxp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:49.294305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.294264    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2e0c5e6-691c-4896-af67-30415a668321-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.294305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.294302    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.294305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.294312    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6rdk\" (UniqueName: \"kubernetes.io/projected/a8d51d22-8ae5-47f6-bd4c-b165017f91b4-kube-api-access-j6rdk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.294559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.294323    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvxp4\" (UniqueName: \"kubernetes.io/projected/c2e0c5e6-691c-4896-af67-30415a668321-kube-api-access-vvxp4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:49.900665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.900564    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"
Oct 31 03:27:49.900665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.900592    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb" event={"ID":"a8d51d22-8ae5-47f6-bd4c-b165017f91b4","Type":"ContainerDied","Data":"e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928"}
Oct 31 03:27:49.900665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.900663    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8cc761f53b8ea7820202c4c587fd91dc0b55ddbcc38846e3c17f7a2d8450928"
Oct 31 03:27:49.902032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.902014    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"
Oct 31 03:27:49.902128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.902055    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq" event={"ID":"c2e0c5e6-691c-4896-af67-30415a668321","Type":"ContainerDied","Data":"979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709"}
Oct 31 03:27:49.902128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:49.902098    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979dbf0bb8ea1c2e0064805072ba5b668ebcdf3b5ab21870111d6b899a45e709"
Oct 31 03:27:53.323707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.323671    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"]
Oct 31 03:27:53.324464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324444    5098 cpu_manager.go:414] "containerMap: removing container" podUID="a8d51d22-8ae5-47f6-bd4c-b165017f91b4" containerName="revoke-keys"
Oct 31 03:27:53.324510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324466    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d51d22-8ae5-47f6-bd4c-b165017f91b4" containerName="revoke-keys"
Oct 31 03:27:53.324510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324491    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" containerName="revoke-keys"
Oct 31 03:27:53.324510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324496    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" containerName="revoke-keys"
Oct 31 03:27:53.324510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324510    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c2e0c5e6-691c-4896-af67-30415a668321" containerName="revoke-keys"
Oct 31 03:27:53.324649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324516    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2e0c5e6-691c-4896-af67-30415a668321" containerName="revoke-keys"
Oct 31 03:27:53.324649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324624    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d51d22-8ae5-47f6-bd4c-b165017f91b4" containerName="revoke-keys"
Oct 31 03:27:53.324649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324635    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" containerName="revoke-keys"
Oct 31 03:27:53.324649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.324642    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2e0c5e6-691c-4896-af67-30415a668321" containerName="revoke-keys"
Oct 31 03:27:53.328347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.328327    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.338118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.338087    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:53.338257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.338121    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:27:53.338257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.338161    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:53.338337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.338294    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:53.343332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.343306    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"]
Oct 31 03:27:53.440461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.440422    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99lql\" (UniqueName: \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.440658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.440573    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.440738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.440699    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.541846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.541811    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.542030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.541866    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.542030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.541954    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99lql\" (UniqueName: \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.542470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.542447    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.544370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.544346    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.550906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.550880    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99lql\" (UniqueName: \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\") pod \"maas-api-revoke-sub-89d8e9f8-pxpv2\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.648401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.648309    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:53.929135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:53.929104    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"]
Oct 31 03:27:53.930105 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:53.930066    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb43f664_561a_4ede_8253_e544b2ee7b0f.slice/crio-6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb WatchSource:0}: Error finding container 6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb: Status 404 returned error can't find the container with id 6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb
Oct 31 03:27:54.939186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:54.939152    5098 generic.go:356] "Generic (PLEG): container finished" podID="cb43f664-561a-4ede-8253-e544b2ee7b0f" containerID="962604d8914430c02fc234c3c48b80d195ef36ac8939fffd2b6401a997fa6455" exitCode=0
Oct 31 03:27:54.939599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:54.939238    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2" event={"ID":"cb43f664-561a-4ede-8253-e544b2ee7b0f","Type":"ContainerDied","Data":"962604d8914430c02fc234c3c48b80d195ef36ac8939fffd2b6401a997fa6455"}
Oct 31 03:27:54.939599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:54.939292    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2" event={"ID":"cb43f664-561a-4ede-8253-e544b2ee7b0f","Type":"ContainerStarted","Data":"6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb"}
Oct 31 03:27:56.111095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.111059    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:56.173582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.173549    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\" (UniqueName: \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\") pod \"cb43f664-561a-4ede-8253-e544b2ee7b0f\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") "
Oct 31 03:27:56.173813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.173638    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\") pod \"cb43f664-561a-4ede-8253-e544b2ee7b0f\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") "
Oct 31 03:27:56.173813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.173714    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\") pod \"cb43f664-561a-4ede-8253-e544b2ee7b0f\" (UID: \"cb43f664-561a-4ede-8253-e544b2ee7b0f\") "
Oct 31 03:27:56.174093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.174061    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca" pod "cb43f664-561a-4ede-8253-e544b2ee7b0f" (UID: "cb43f664-561a-4ede-8253-e544b2ee7b0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:27:56.174302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.174275    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:56.176216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.176184    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca" pod "cb43f664-561a-4ede-8253-e544b2ee7b0f" (UID: "cb43f664-561a-4ede-8253-e544b2ee7b0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:27:56.176711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.176685    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql" pod "cb43f664-561a-4ede-8253-e544b2ee7b0f" (UID: "cb43f664-561a-4ede-8253-e544b2ee7b0f"). InnerVolumeSpecName "kube-api-access-99lql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:27:56.255540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.255453    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:27:56.275706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.275672    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb43f664-561a-4ede-8253-e544b2ee7b0f-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:56.275706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.275704    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99lql\" (UniqueName: \"kubernetes.io/projected/cb43f664-561a-4ede-8253-e544b2ee7b0f-kube-api-access-99lql\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:27:56.956642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.956589    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"
Oct 31 03:27:56.956819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.956600    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2" event={"ID":"cb43f664-561a-4ede-8253-e544b2ee7b0f","Type":"ContainerDied","Data":"6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb"}
Oct 31 03:27:56.956819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:56.956689    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f097e43c7474a52c03b64b4cec839428ff199d1926cb0793c952ccb6c2671bb"
Oct 31 03:27:58.296286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.296240    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"]
Oct 31 03:27:58.296806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.296788    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cb43f664-561a-4ede-8253-e544b2ee7b0f" containerName="revoke-keys"
Oct 31 03:27:58.296850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.296809    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb43f664-561a-4ede-8253-e544b2ee7b0f" containerName="revoke-keys"
Oct 31 03:27:58.297034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.297020    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb43f664-561a-4ede-8253-e544b2ee7b0f" containerName="revoke-keys"
Oct 31 03:27:58.300839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.300820    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.305543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.305509    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:27:58.305718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.305581    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:27:58.306837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.306809    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:27:58.306987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.306968    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:27:58.311534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.311508    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"]
Oct 31 03:27:58.413223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.413185    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.413424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.413357    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hlrh\" (UniqueName: \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.413471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.413458    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.520036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.516022    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.520036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.516183    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.520036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.516392    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hlrh\" (UniqueName: \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.520036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.518205    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.520036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.519601    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.527931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.527903    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hlrh\" (UniqueName: \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\") pod \"maas-api-revoke-sub-34613fef-4brmm\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.617721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.617597    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:27:58.666473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.666438    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"]
Oct 31 03:27:58.671558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.671531    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.688367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.688264    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"]
Oct 31 03:27:58.819977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.819940    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.820165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.820097    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljdqf\" (UniqueName: \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.820220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.820197    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.909935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.909907    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"]
Oct 31 03:27:58.910735 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:58.910710    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481ab210_f02a_42dc_abba_28979b89e349.slice/crio-3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81 WatchSource:0}: Error finding container 3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81: Status 404 returned error can't find the container with id 3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81
Oct 31 03:27:58.921979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.921942    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljdqf\" (UniqueName: \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.922119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.922023    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.922119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.922082    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.922910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.922885    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.924454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.924436    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.930560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.930532    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljdqf\" (UniqueName: \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\") pod \"maas-api-revoke-sub-79668691-z75hj\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:58.973586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.973554    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm" event={"ID":"481ab210-f02a-42dc-abba-28979b89e349","Type":"ContainerStarted","Data":"3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81"}
Oct 31 03:27:58.994806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:58.994764    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:27:59.283409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:59.283341    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"]
Oct 31 03:27:59.292751 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:27:59.292710    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac0a529_81e4_4a96_94d7_49a853da4c8b.slice/crio-e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34 WatchSource:0}: Error finding container e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34: Status 404 returned error can't find the container with id e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34
Oct 31 03:27:59.981261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:59.981226    5098 generic.go:356] "Generic (PLEG): container finished" podID="481ab210-f02a-42dc-abba-28979b89e349" containerID="aad0b8af2d60df47b483fbcbe4497cf5debc1ff1f594b444dc689a0f80c3eb6a" exitCode=0
Oct 31 03:27:59.981722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:59.981316    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm" event={"ID":"481ab210-f02a-42dc-abba-28979b89e349","Type":"ContainerDied","Data":"aad0b8af2d60df47b483fbcbe4497cf5debc1ff1f594b444dc689a0f80c3eb6a"}
Oct 31 03:27:59.983045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:59.983016    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj" event={"ID":"1ac0a529-81e4-4a96-94d7-49a853da4c8b","Type":"ContainerStarted","Data":"8fd67211f53ee8a0b0d2b6f3397ef20a470efee330d53695d6af0ce10e05a585"}
Oct 31 03:27:59.983163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:27:59.983064    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj" event={"ID":"1ac0a529-81e4-4a96-94d7-49a853da4c8b","Type":"ContainerStarted","Data":"e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34"}
Oct 31 03:28:00.029904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:00.029841    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj" podStartSLOduration=2.029824379 podStartE2EDuration="2.029824379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:00.027962048 +0000 UTC m=+754.553181640" watchObservedRunningTime="2026-10-31 03:28:00.029824379 +0000 UTC m=+754.555043969"
Oct 31 03:28:00.991150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:00.991114    5098 generic.go:356] "Generic (PLEG): container finished" podID="1ac0a529-81e4-4a96-94d7-49a853da4c8b" containerID="8fd67211f53ee8a0b0d2b6f3397ef20a470efee330d53695d6af0ce10e05a585" exitCode=0
Oct 31 03:28:00.991751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:00.991196    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj" event={"ID":"1ac0a529-81e4-4a96-94d7-49a853da4c8b","Type":"ContainerDied","Data":"8fd67211f53ee8a0b0d2b6f3397ef20a470efee330d53695d6af0ce10e05a585"}
Oct 31 03:28:01.062535 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:28:01.062491    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:28:01.187336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.187305    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:28:01.250671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.250551    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\" (UniqueName: \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\") pod \"481ab210-f02a-42dc-abba-28979b89e349\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") "
Oct 31 03:28:01.250671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.250595    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\") pod \"481ab210-f02a-42dc-abba-28979b89e349\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") "
Oct 31 03:28:01.250899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.250704    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\") pod \"481ab210-f02a-42dc-abba-28979b89e349\" (UID: \"481ab210-f02a-42dc-abba-28979b89e349\") "
Oct 31 03:28:01.251081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.251052    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca" pod "481ab210-f02a-42dc-abba-28979b89e349" (UID: "481ab210-f02a-42dc-abba-28979b89e349"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:01.252871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.252844    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca" pod "481ab210-f02a-42dc-abba-28979b89e349" (UID: "481ab210-f02a-42dc-abba-28979b89e349"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:01.252968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.252868    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh" pod "481ab210-f02a-42dc-abba-28979b89e349" (UID: "481ab210-f02a-42dc-abba-28979b89e349"). InnerVolumeSpecName "kube-api-access-5hlrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:01.372781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.369204    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hlrh\" (UniqueName: \"kubernetes.io/projected/481ab210-f02a-42dc-abba-28979b89e349-kube-api-access-5hlrh\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:01.372781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.369251    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481ab210-f02a-42dc-abba-28979b89e349-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:01.372781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.369276    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481ab210-f02a-42dc-abba-28979b89e349-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:01.836234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.836189    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"]
Oct 31 03:28:01.836868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.836846    5098 cpu_manager.go:414] "containerMap: removing container" podUID="481ab210-f02a-42dc-abba-28979b89e349" containerName="revoke-keys"
Oct 31 03:28:01.836868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.836867    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481ab210-f02a-42dc-abba-28979b89e349" containerName="revoke-keys"
Oct 31 03:28:01.837038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.836991    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="481ab210-f02a-42dc-abba-28979b89e349" containerName="revoke-keys"
Oct 31 03:28:01.840647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.840594    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:01.849470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.849439    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"]
Oct 31 03:28:01.974844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.974743    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:01.974844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.974798    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:01.975087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:01.974937    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdccb\" (UniqueName: \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.000236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.000201    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"
Oct 31 03:28:02.000236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.000201    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm" event={"ID":"481ab210-f02a-42dc-abba-28979b89e349","Type":"ContainerDied","Data":"3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81"}
Oct 31 03:28:02.000681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.000274    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e72b4d0a6903e451ba4a4214e355e29749c38d9b536ca14467c1e7943db8f81"
Oct 31 03:28:02.076802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.076759    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdccb\" (UniqueName: \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.077003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.076984    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.077067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.077036    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.077877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.077852    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.082352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.082320    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.095311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.095269    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdccb\" (UniqueName: \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\") pod \"maas-api-revoke-sub-541e497c-sn9wh\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.136301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.136270    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:28:02.157684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.157651    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:02.279205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.279167    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\") pod \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") "
Oct 31 03:28:02.279409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.279293    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\") pod \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") "
Oct 31 03:28:02.279409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.279395    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\" (UniqueName: \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\") pod \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\" (UID: \"1ac0a529-81e4-4a96-94d7-49a853da4c8b\") "
Oct 31 03:28:02.279708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.279674    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca" pod "1ac0a529-81e4-4a96-94d7-49a853da4c8b" (UID: "1ac0a529-81e4-4a96-94d7-49a853da4c8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:02.280022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.279984    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:02.282069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.282040    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf" pod "1ac0a529-81e4-4a96-94d7-49a853da4c8b" (UID: "1ac0a529-81e4-4a96-94d7-49a853da4c8b"). InnerVolumeSpecName "kube-api-access-ljdqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:02.282069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.282051    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca" pod "1ac0a529-81e4-4a96-94d7-49a853da4c8b" (UID: "1ac0a529-81e4-4a96-94d7-49a853da4c8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:02.381271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.381232    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljdqf\" (UniqueName: \"kubernetes.io/projected/1ac0a529-81e4-4a96-94d7-49a853da4c8b-kube-api-access-ljdqf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:02.381271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.381266    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ac0a529-81e4-4a96-94d7-49a853da4c8b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:02.446549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:02.446508    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"]
Oct 31 03:28:02.446760 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:02.446729    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode633d3fc_b0d4_4b70_83ec_9e578b66772a.slice/crio-4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a WatchSource:0}: Error finding container 4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a: Status 404 returned error can't find the container with id 4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a
Oct 31 03:28:03.008940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.008901    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh" event={"ID":"e633d3fc-b0d4-4b70-83ec-9e578b66772a","Type":"ContainerStarted","Data":"6bc8298acb22bb4715b6f4032ff764492626083528e27bedc00cc16a2b8b6493"}
Oct 31 03:28:03.009399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.008998    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh" event={"ID":"e633d3fc-b0d4-4b70-83ec-9e578b66772a","Type":"ContainerStarted","Data":"4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a"}
Oct 31 03:28:03.010372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.010342    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj" event={"ID":"1ac0a529-81e4-4a96-94d7-49a853da4c8b","Type":"ContainerDied","Data":"e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34"}
Oct 31 03:28:03.010372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.010360    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"
Oct 31 03:28:03.010534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.010390    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e75997a3ad5d921cb262de8ed2cc05664c222c5839b1a2c520676ba2046baf34"
Oct 31 03:28:03.047901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:03.047825    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh" podStartSLOduration=2.04780063 podStartE2EDuration="2.04780063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:03.040015449 +0000 UTC m=+757.565235038" watchObservedRunningTime="2026-10-31 03:28:03.04780063 +0000 UTC m=+757.573020220"
Oct 31 03:28:04.018093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.018059    5098 generic.go:356] "Generic (PLEG): container finished" podID="e633d3fc-b0d4-4b70-83ec-9e578b66772a" containerID="6bc8298acb22bb4715b6f4032ff764492626083528e27bedc00cc16a2b8b6493" exitCode=0
Oct 31 03:28:04.018561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.018114    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh" event={"ID":"e633d3fc-b0d4-4b70-83ec-9e578b66772a","Type":"ContainerDied","Data":"6bc8298acb22bb4715b6f4032ff764492626083528e27bedc00cc16a2b8b6493"}
Oct 31 03:28:04.183548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.183508    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"]
Oct 31 03:28:04.184088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.184074    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1ac0a529-81e4-4a96-94d7-49a853da4c8b" containerName="revoke-keys"
Oct 31 03:28:04.184147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.184090    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac0a529-81e4-4a96-94d7-49a853da4c8b" containerName="revoke-keys"
Oct 31 03:28:04.184199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.184190    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac0a529-81e4-4a96-94d7-49a853da4c8b" containerName="revoke-keys"
Oct 31 03:28:04.188204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.188186    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.197258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.197224    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"]
Oct 31 03:28:04.308746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.308706    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.308975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.308775    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.308975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.308906    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sh9w\" (UniqueName: \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.410360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.410322    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sh9w\" (UniqueName: \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.410515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.410464    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.410575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.410512    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.411190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.411167    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.413064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.413034    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.419000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.418974    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sh9w\" (UniqueName: \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\") pod \"maas-api-revoke-sub-3cf07860-2snv9\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.507501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.507468    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:04.790058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:04.790024    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"]
Oct 31 03:28:04.791174 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:04.791135    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ea8b7a_b485_4435_954e_d38b718297b1.slice/crio-8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233 WatchSource:0}: Error finding container 8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233: Status 404 returned error can't find the container with id 8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233
Oct 31 03:28:05.026352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.026315    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9" event={"ID":"29ea8b7a-b485-4435-954e-d38b718297b1","Type":"ContainerStarted","Data":"8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233"}
Oct 31 03:28:05.161995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.161971    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:05.219404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.219370    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\") pod \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") "
Oct 31 03:28:05.219586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.219472    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\") pod \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") "
Oct 31 03:28:05.219586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.219554    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\" (UniqueName: \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\") pod \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\" (UID: \"e633d3fc-b0d4-4b70-83ec-9e578b66772a\") "
Oct 31 03:28:05.219873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.219844    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca" pod "e633d3fc-b0d4-4b70-83ec-9e578b66772a" (UID: "e633d3fc-b0d4-4b70-83ec-9e578b66772a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:05.221999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.221972    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb" pod "e633d3fc-b0d4-4b70-83ec-9e578b66772a" (UID: "e633d3fc-b0d4-4b70-83ec-9e578b66772a"). InnerVolumeSpecName "kube-api-access-zdccb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:05.222138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.222012    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca" pod "e633d3fc-b0d4-4b70-83ec-9e578b66772a" (UID: "e633d3fc-b0d4-4b70-83ec-9e578b66772a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:05.321126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.321091    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:05.321126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.321122    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e633d3fc-b0d4-4b70-83ec-9e578b66772a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:05.321345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:05.321133    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdccb\" (UniqueName: \"kubernetes.io/projected/e633d3fc-b0d4-4b70-83ec-9e578b66772a-kube-api-access-zdccb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:06.034078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.034042    5098 generic.go:356] "Generic (PLEG): container finished" podID="29ea8b7a-b485-4435-954e-d38b718297b1" containerID="b047d347fc5555c4b239435cd10987676ce2620eccb1ffbe9daec4bfb50c62bd" exitCode=0
Oct 31 03:28:06.034527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.034126    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9" event={"ID":"29ea8b7a-b485-4435-954e-d38b718297b1","Type":"ContainerDied","Data":"b047d347fc5555c4b239435cd10987676ce2620eccb1ffbe9daec4bfb50c62bd"}
Oct 31 03:28:06.035537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.035517    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"
Oct 31 03:28:06.035703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.035544    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh" event={"ID":"e633d3fc-b0d4-4b70-83ec-9e578b66772a","Type":"ContainerDied","Data":"4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a"}
Oct 31 03:28:06.035703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.035582    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4f5baf34f016e491e2a11ef422c86c722c3be30861034fae9505e37b8c174a"
Oct 31 03:28:06.928506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.928468    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"]
Oct 31 03:28:06.929046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.929030    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e633d3fc-b0d4-4b70-83ec-9e578b66772a" containerName="revoke-keys"
Oct 31 03:28:06.929046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.929046    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e633d3fc-b0d4-4b70-83ec-9e578b66772a" containerName="revoke-keys"
Oct 31 03:28:06.929197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.929159    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e633d3fc-b0d4-4b70-83ec-9e578b66772a" containerName="revoke-keys"
Oct 31 03:28:06.933006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.932982    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:06.940757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:06.940729    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"]
Oct 31 03:28:07.052379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.052341    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.052782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.052412    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.052782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.052657    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dgh\" (UniqueName: \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.154456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.154413    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dgh\" (UniqueName: \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.154661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.154516    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.154661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.154567    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.155262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.155238    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.157273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.157239    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.163365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.163336    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dgh\" (UniqueName: \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\") pod \"maas-api-revoke-sub-3a2f9357-874qg\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.209378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.209348    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:07.252971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.252927    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:07.358055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.357996    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\") pod \"29ea8b7a-b485-4435-954e-d38b718297b1\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") "
Oct 31 03:28:07.358302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.358277    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\") pod \"29ea8b7a-b485-4435-954e-d38b718297b1\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") "
Oct 31 03:28:07.358391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.358330    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\" (UniqueName: \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\") pod \"29ea8b7a-b485-4435-954e-d38b718297b1\" (UID: \"29ea8b7a-b485-4435-954e-d38b718297b1\") "
Oct 31 03:28:07.359896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.359870    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca" pod "29ea8b7a-b485-4435-954e-d38b718297b1" (UID: "29ea8b7a-b485-4435-954e-d38b718297b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:07.362385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.362361    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w" pod "29ea8b7a-b485-4435-954e-d38b718297b1" (UID: "29ea8b7a-b485-4435-954e-d38b718297b1"). InnerVolumeSpecName "kube-api-access-9sh9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:07.366433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.366402    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca" pod "29ea8b7a-b485-4435-954e-d38b718297b1" (UID: "29ea8b7a-b485-4435-954e-d38b718297b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:07.460559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.460463    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:07.460559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.460502    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sh9w\" (UniqueName: \"kubernetes.io/projected/29ea8b7a-b485-4435-954e-d38b718297b1-kube-api-access-9sh9w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:07.460559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.460513    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29ea8b7a-b485-4435-954e-d38b718297b1-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:07.545681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:07.545648    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"]
Oct 31 03:28:07.546851 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:07.546820    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61abb82_737f_471c_b399_500515b70c1b.slice/crio-7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b WatchSource:0}: Error finding container 7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b: Status 404 returned error can't find the container with id 7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b
Oct 31 03:28:08.053530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.053474    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg" event={"ID":"d61abb82-737f-471c-b399-500515b70c1b","Type":"ContainerStarted","Data":"32cff6623d858deb2cedc277cc4fe8f5d17b78a25285684c60d56dac57e153a0"}
Oct 31 03:28:08.053970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.053554    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg" event={"ID":"d61abb82-737f-471c-b399-500515b70c1b","Type":"ContainerStarted","Data":"7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b"}
Oct 31 03:28:08.055209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.055182    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9" event={"ID":"29ea8b7a-b485-4435-954e-d38b718297b1","Type":"ContainerDied","Data":"8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233"}
Oct 31 03:28:08.055209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.055207    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"
Oct 31 03:28:08.055423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.055224    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8901f9ea720e6753ef1110eead6a1fb6a079c70e8a22322e782416f7683d5233"
Oct 31 03:28:08.088631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:08.077726    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg" podStartSLOduration=2.077690682 podStartE2EDuration="2.077690682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:08.07175069 +0000 UTC m=+762.596970280" watchObservedRunningTime="2026-10-31 03:28:08.077690682 +0000 UTC m=+762.602910273"
Oct 31 03:28:09.071804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:09.071756    5098 generic.go:356] "Generic (PLEG): container finished" podID="d61abb82-737f-471c-b399-500515b70c1b" containerID="32cff6623d858deb2cedc277cc4fe8f5d17b78a25285684c60d56dac57e153a0" exitCode=0
Oct 31 03:28:09.072533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:09.072503    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg" event={"ID":"d61abb82-737f-471c-b399-500515b70c1b","Type":"ContainerDied","Data":"32cff6623d858deb2cedc277cc4fe8f5d17b78a25285684c60d56dac57e153a0"}
Oct 31 03:28:10.628823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.628789    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:10.744204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.744098    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\") pod \"d61abb82-737f-471c-b399-500515b70c1b\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") "
Oct 31 03:28:10.744574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.744538    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\") pod \"d61abb82-737f-471c-b399-500515b70c1b\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") "
Oct 31 03:28:10.745827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.745224    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\" (UniqueName: \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\") pod \"d61abb82-737f-471c-b399-500515b70c1b\" (UID: \"d61abb82-737f-471c-b399-500515b70c1b\") "
Oct 31 03:28:10.745827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.745409    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca" pod "d61abb82-737f-471c-b399-500515b70c1b" (UID: "d61abb82-737f-471c-b399-500515b70c1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:10.746703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.746678    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d61abb82-737f-471c-b399-500515b70c1b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:10.748205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.748181    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca" pod "d61abb82-737f-471c-b399-500515b70c1b" (UID: "d61abb82-737f-471c-b399-500515b70c1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:10.748324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.748187    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh" pod "d61abb82-737f-471c-b399-500515b70c1b" (UID: "d61abb82-737f-471c-b399-500515b70c1b"). InnerVolumeSpecName "kube-api-access-z2dgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:10.848582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.848539    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d61abb82-737f-471c-b399-500515b70c1b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:10.848582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:10.848585    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dgh\" (UniqueName: \"kubernetes.io/projected/d61abb82-737f-471c-b399-500515b70c1b-kube-api-access-z2dgh\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:11.109346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:11.106702    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"
Oct 31 03:28:11.109346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:11.106731    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg" event={"ID":"d61abb82-737f-471c-b399-500515b70c1b","Type":"ContainerDied","Data":"7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b"}
Oct 31 03:28:11.109346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:11.106821    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc29f1da76fe6e9c76d9a0759ceec738bd0a3c0339cb8612225582ba758bb7b"
Oct 31 03:28:12.864840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.864793    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"]
Oct 31 03:28:12.865706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865676    5098 cpu_manager.go:414] "containerMap: removing container" podUID="29ea8b7a-b485-4435-954e-d38b718297b1" containerName="revoke-keys"
Oct 31 03:28:12.865706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865704    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ea8b7a-b485-4435-954e-d38b718297b1" containerName="revoke-keys"
Oct 31 03:28:12.865948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865746    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d61abb82-737f-471c-b399-500515b70c1b" containerName="revoke-keys"
Oct 31 03:28:12.865948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865755    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d61abb82-737f-471c-b399-500515b70c1b" containerName="revoke-keys"
Oct 31 03:28:12.866077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865965    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ea8b7a-b485-4435-954e-d38b718297b1" containerName="revoke-keys"
Oct 31 03:28:12.866077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.865982    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d61abb82-737f-471c-b399-500515b70c1b" containerName="revoke-keys"
Oct 31 03:28:12.870185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.870158    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:12.874887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.874862    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:12.876312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.876287    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:12.877192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.876654    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:12.877192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.876815    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:28:12.877192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.876923    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"]
Oct 31 03:28:12.980198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.980158    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:12.980381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.980345    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mplw8\" (UniqueName: \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:12.980429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:12.980411    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.082124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.082081    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.082303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.082202    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mplw8\" (UniqueName: \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.082303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.082250    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.082918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.082898    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.084652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.084631    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.090614 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.090585    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mplw8\" (UniqueName: \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\") pod \"maas-api-revoke-sub-41536206-gq56c\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.197469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.197380    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:13.481146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:13.481117    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"]
Oct 31 03:28:13.481947 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:13.481918    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e9e506_4d3f_4b86_ad1d_d62b48e4542f.slice/crio-aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e WatchSource:0}: Error finding container aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e: Status 404 returned error can't find the container with id aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e
Oct 31 03:28:14.091916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.091883    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:28:14.095919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.095897    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.105679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.105650    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"distinct-w6-7f3159-kserve-self-signed-certs\""
Oct 31 03:28:14.105893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.105868    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"openshift-service-ca.crt\""
Oct 31 03:28:14.106023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.105788    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"kube-root-ca.crt\""
Oct 31 03:28:14.106131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.105731    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-7f3159\"/\"default-dockercfg-b78xs\""
Oct 31 03:28:14.109406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.107740    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:28:14.131850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.131805    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c" event={"ID":"75e9e506-4d3f-4b86-ad1d-d62b48e4542f","Type":"ContainerStarted","Data":"93f88c300af62fd5789d161ac218e3629f7328b9d883b7dc552d5593cf15ffff"}
Oct 31 03:28:14.132009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.131904    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c" event={"ID":"75e9e506-4d3f-4b86-ad1d-d62b48e4542f","Type":"ContainerStarted","Data":"aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e"}
Oct 31 03:28:14.157893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.157829    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c" podStartSLOduration=2.157812478 podStartE2EDuration="2.157812478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:14.154583467 +0000 UTC m=+768.679803035" watchObservedRunningTime="2026-10-31 03:28:14.157812478 +0000 UTC m=+768.683032066"
Oct 31 03:28:14.197047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.196680    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.197047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.196810    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.197047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.196901    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.197047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.196935    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.197047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.196963    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.197438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.197229    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssmrw\" (UniqueName: \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298343    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298385    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298449    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssmrw\" (UniqueName: \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298635    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298669    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298718    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298813    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.298899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298884    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.299015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.298999    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.300873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.300849    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.301172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.301152    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.306586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.306563    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssmrw\" (UniqueName: \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\") pod \"distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") " pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.423672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.423601    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:14.705644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:14.705601    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:28:14.706553 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:14.706530    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8554b90_54d5_4f2a_aa1c_5c5d0644495b.slice/crio-267b471cc56b9fffe9a811e5090f1d33b030f4adcf27ae1009db88e9639add1d WatchSource:0}: Error finding container 267b471cc56b9fffe9a811e5090f1d33b030f4adcf27ae1009db88e9639add1d: Status 404 returned error can't find the container with id 267b471cc56b9fffe9a811e5090f1d33b030f4adcf27ae1009db88e9639add1d
Oct 31 03:28:15.125370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.125335    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"]
Oct 31 03:28:15.129733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.129704    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.143652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.138056    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"]
Oct 31 03:28:15.144439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.144407    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" event={"ID":"b8554b90-54d5-4f2a-aa1c-5c5d0644495b","Type":"ContainerStarted","Data":"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"}
Oct 31 03:28:15.144601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.144465    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" event={"ID":"b8554b90-54d5-4f2a-aa1c-5c5d0644495b","Type":"ContainerStarted","Data":"267b471cc56b9fffe9a811e5090f1d33b030f4adcf27ae1009db88e9639add1d"}
Oct 31 03:28:15.147662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.147630    5098 generic.go:356] "Generic (PLEG): container finished" podID="75e9e506-4d3f-4b86-ad1d-d62b48e4542f" containerID="93f88c300af62fd5789d161ac218e3629f7328b9d883b7dc552d5593cf15ffff" exitCode=0
Oct 31 03:28:15.147767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.147691    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c" event={"ID":"75e9e506-4d3f-4b86-ad1d-d62b48e4542f","Type":"ContainerDied","Data":"93f88c300af62fd5789d161ac218e3629f7328b9d883b7dc552d5593cf15ffff"}
Oct 31 03:28:15.167653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.167581    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podStartSLOduration=1.167564029 podStartE2EDuration="1.167564029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:15.165922889 +0000 UTC m=+769.691142480" watchObservedRunningTime="2026-10-31 03:28:15.167564029 +0000 UTC m=+769.692783617"
Oct 31 03:28:15.211141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.211094    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.211324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.211247    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkqz\" (UniqueName: \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.211413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.211327    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.312409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.312365    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.312661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.312553    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.312752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.312664    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkqz\" (UniqueName: \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.313147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.313128    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.315007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.314988    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.322580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.322551    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkqz\" (UniqueName: \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\") pod \"maas-api-revoke-sub-6e26e762-b6w2c\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.424862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.424777    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:15.458796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.458762    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:15.750663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:15.750630    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"]
Oct 31 03:28:15.751276 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:15.751245    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6df435_a0c8_4d94_94aa_570c4f3e5936.slice/crio-ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92 WatchSource:0}: Error finding container ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92: Status 404 returned error can't find the container with id ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92
Oct 31 03:28:16.156282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.156231    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c" event={"ID":"5e6df435-a0c8-4d94-94aa-570c4f3e5936","Type":"ContainerStarted","Data":"fa96b8f461a326f53398bdf27b1d95a8166731bb70c17c778703f009da5d101d"}
Oct 31 03:28:16.156803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.156313    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c" event={"ID":"5e6df435-a0c8-4d94-94aa-570c4f3e5936","Type":"ContainerStarted","Data":"ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92"}
Oct 31 03:28:16.176161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.176105    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c" podStartSLOduration=1.176089805 podStartE2EDuration="1.176089805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:16.17514951 +0000 UTC m=+770.700369102" watchObservedRunningTime="2026-10-31 03:28:16.176089805 +0000 UTC m=+770.701309389"
Oct 31 03:28:16.316425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.316346    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:16.406886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.406854    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"]
Oct 31 03:28:16.407409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.407397    5098 cpu_manager.go:414] "containerMap: removing container" podUID="75e9e506-4d3f-4b86-ad1d-d62b48e4542f" containerName="revoke-keys"
Oct 31 03:28:16.407453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.407411    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e9e506-4d3f-4b86-ad1d-d62b48e4542f" containerName="revoke-keys"
Oct 31 03:28:16.407524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.407515    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e9e506-4d3f-4b86-ad1d-d62b48e4542f" containerName="revoke-keys"
Oct 31 03:28:16.410862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.410844    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.419268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.419237    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"]
Oct 31 03:28:16.428295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.428267    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\" (UniqueName: \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\") pod \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") "
Oct 31 03:28:16.428440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.428354    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\") pod \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") "
Oct 31 03:28:16.428521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.428508    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\") pod \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\" (UID: \"75e9e506-4d3f-4b86-ad1d-d62b48e4542f\") "
Oct 31 03:28:16.431147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.431118    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca" pod "75e9e506-4d3f-4b86-ad1d-d62b48e4542f" (UID: "75e9e506-4d3f-4b86-ad1d-d62b48e4542f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:16.431397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.431377    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca" pod "75e9e506-4d3f-4b86-ad1d-d62b48e4542f" (UID: "75e9e506-4d3f-4b86-ad1d-d62b48e4542f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:16.432351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.432325    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8" pod "75e9e506-4d3f-4b86-ad1d-d62b48e4542f" (UID: "75e9e506-4d3f-4b86-ad1d-d62b48e4542f"). InnerVolumeSpecName "kube-api-access-mplw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:16.529584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.529547    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.529811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.529734    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.529891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.529864    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq6pg\" (UniqueName: \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.529987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.529973    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:16.530027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.529999    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mplw8\" (UniqueName: \"kubernetes.io/projected/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-kube-api-access-mplw8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:16.530027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.530015    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75e9e506-4d3f-4b86-ad1d-d62b48e4542f-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:16.630903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.630866    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq6pg\" (UniqueName: \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.631014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.630990    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.631064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.631049    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.631780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.631760    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.633474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.633451    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.641370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.641346    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq6pg\" (UniqueName: \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\") pod \"maas-api-revoke-sub-219891e1-l2vtg\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:16.728548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:16.728507    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:17.005892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.005859    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"]
Oct 31 03:28:17.006695 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:17.006659    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb542a9bd_4c75_437f_a939_559b47450e6a.slice/crio-5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc WatchSource:0}: Error finding container 5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc: Status 404 returned error can't find the container with id 5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc
Oct 31 03:28:17.167526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.167498    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"
Oct 31 03:28:17.167954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.167495    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c" event={"ID":"75e9e506-4d3f-4b86-ad1d-d62b48e4542f","Type":"ContainerDied","Data":"aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e"}
Oct 31 03:28:17.167954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.167666    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aeb2ff7012e12ff38a1aa6c1a112b147fcf315aadab7df477d1a7cdc8e84ac9e"
Oct 31 03:28:17.168982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.168955    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg" event={"ID":"b542a9bd-4c75-437f-a939-559b47450e6a","Type":"ContainerStarted","Data":"5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc"}
Oct 31 03:28:17.170528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.170503    5098 generic.go:356] "Generic (PLEG): container finished" podID="5e6df435-a0c8-4d94-94aa-570c4f3e5936" containerID="fa96b8f461a326f53398bdf27b1d95a8166731bb70c17c778703f009da5d101d" exitCode=0
Oct 31 03:28:17.170639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.170584    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c" event={"ID":"5e6df435-a0c8-4d94-94aa-570c4f3e5936","Type":"ContainerDied","Data":"fa96b8f461a326f53398bdf27b1d95a8166731bb70c17c778703f009da5d101d"}
Oct 31 03:28:17.276484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.276453    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"]
Oct 31 03:28:17.282217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.282186    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.286679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.286649    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"]
Oct 31 03:28:17.340291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.340258    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.340485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.340318    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.340485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.340365    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nddfn\" (UniqueName: \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.442420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.442375    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.442641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.442439    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.442641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.442487    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nddfn\" (UniqueName: \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.443524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.443495    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.444934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.444915    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.452213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.452183    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nddfn\" (UniqueName: \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\") pod \"maas-api-revoke-sub-f81ea3e6-j5p6n\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.608455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.608420    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:17.891382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:17.891352    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"]
Oct 31 03:28:17.891828 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:17.891808    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b59b8_d383_459b_87ad_fa8eaf0cbe63.slice/crio-570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562 WatchSource:0}: Error finding container 570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562: Status 404 returned error can't find the container with id 570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562
Oct 31 03:28:18.179970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.179877    5098 generic.go:356] "Generic (PLEG): container finished" podID="b542a9bd-4c75-437f-a939-559b47450e6a" containerID="52c5f917df0258c40fe3fb984d2b6d4b26ae564fc50650964b378f1fa2e53ad5" exitCode=0
Oct 31 03:28:18.180387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.179972    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg" event={"ID":"b542a9bd-4c75-437f-a939-559b47450e6a","Type":"ContainerDied","Data":"52c5f917df0258c40fe3fb984d2b6d4b26ae564fc50650964b378f1fa2e53ad5"}
Oct 31 03:28:18.181443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.181411    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n" event={"ID":"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63","Type":"ContainerStarted","Data":"570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562"}
Oct 31 03:28:18.506250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.505836    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:18.601067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.601018    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\" (UniqueName: \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\") pod \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") "
Oct 31 03:28:18.601285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.601098    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\") pod \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") "
Oct 31 03:28:18.601285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.601180    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\") pod \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\" (UID: \"5e6df435-a0c8-4d94-94aa-570c4f3e5936\") "
Oct 31 03:28:18.614326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.614286    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca" pod "5e6df435-a0c8-4d94-94aa-570c4f3e5936" (UID: "5e6df435-a0c8-4d94-94aa-570c4f3e5936"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:18.617684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.616812    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz" pod "5e6df435-a0c8-4d94-94aa-570c4f3e5936" (UID: "5e6df435-a0c8-4d94-94aa-570c4f3e5936"). InnerVolumeSpecName "kube-api-access-vlkqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:18.625888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.625850    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca" pod "5e6df435-a0c8-4d94-94aa-570c4f3e5936" (UID: "5e6df435-a0c8-4d94-94aa-570c4f3e5936"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:18.703099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.703057    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlkqz\" (UniqueName: \"kubernetes.io/projected/5e6df435-a0c8-4d94-94aa-570c4f3e5936-kube-api-access-vlkqz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:18.703099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.703101    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:18.703358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.703118    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6df435-a0c8-4d94-94aa-570c4f3e5936-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:18.967022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.966809    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:28:18.968498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.967823    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5e6df435-a0c8-4d94-94aa-570c4f3e5936" containerName="revoke-keys"
Oct 31 03:28:18.968498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.967853    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6df435-a0c8-4d94-94aa-570c4f3e5936" containerName="revoke-keys"
Oct 31 03:28:18.968498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.968058    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6df435-a0c8-4d94-94aa-570c4f3e5936" containerName="revoke-keys"
Oct 31 03:28:18.972263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.972243    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:18.991761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.991432    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:28:18.991761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:18.991668    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-d86a77-openshift-default-dockercfg-2xd8c\""
Oct 31 03:28:19.108391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.108345    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.108772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.108745    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.108991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.108973    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.109185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.109166    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wc84\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.109359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.109342    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.109526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.109510    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.109696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.109679    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.109894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.109877    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.110138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.110118    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.191843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.191796    5098 generic.go:356] "Generic (PLEG): container finished" podID="1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" containerID="c667487176f0c42ee76e5642675eaff77dc9c0f09de4579106fa38853d72d45b" exitCode=0
Oct 31 03:28:19.192316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.191946    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n" event={"ID":"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63","Type":"ContainerDied","Data":"c667487176f0c42ee76e5642675eaff77dc9c0f09de4579106fa38853d72d45b"}
Oct 31 03:28:19.193746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.193716    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c" event={"ID":"5e6df435-a0c8-4d94-94aa-570c4f3e5936","Type":"ContainerDied","Data":"ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92"}
Oct 31 03:28:19.193920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.193903    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee9a92210cff58d2b5043637719d5a78b498c19135b8392ffabc46e56638af92"
Oct 31 03:28:19.194438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.193863    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.211742    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.211828    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.211904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.211978    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wc84\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.212011    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.212054    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.212092    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.212149    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.213530    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.214098    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.216324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.215275    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.219206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.217016    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.219206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.217557    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.219206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.218796    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.231251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.231173    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.235579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.235536    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.236352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.236326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.237622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.237588    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wc84\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.339241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.339202    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:19.391968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.391670    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:19.520498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.520405    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\") pod \"b542a9bd-4c75-437f-a939-559b47450e6a\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") "
Oct 31 03:28:19.520895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.520871    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\" (UniqueName: \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\") pod \"b542a9bd-4c75-437f-a939-559b47450e6a\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") "
Oct 31 03:28:19.521081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.521063    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\") pod \"b542a9bd-4c75-437f-a939-559b47450e6a\" (UID: \"b542a9bd-4c75-437f-a939-559b47450e6a\") "
Oct 31 03:28:19.521784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.521751    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca" pod "b542a9bd-4c75-437f-a939-559b47450e6a" (UID: "b542a9bd-4c75-437f-a939-559b47450e6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:19.523202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.523177    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg" pod "b542a9bd-4c75-437f-a939-559b47450e6a" (UID: "b542a9bd-4c75-437f-a939-559b47450e6a"). InnerVolumeSpecName "kube-api-access-sq6pg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:19.523403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.523375    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca" pod "b542a9bd-4c75-437f-a939-559b47450e6a" (UID: "b542a9bd-4c75-437f-a939-559b47450e6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:19.622369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.622335    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:19.622369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.622367    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b542a9bd-4c75-437f-a939-559b47450e6a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:19.622564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.622379    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq6pg\" (UniqueName: \"kubernetes.io/projected/b542a9bd-4c75-437f-a939-559b47450e6a-kube-api-access-sq6pg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:19.650703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.650659    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:28:19.652919 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:19.652882    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9154c52d_f1a1_45a0_803a_c3384b81c35d.slice/crio-6d15ba4503b213e32bcaf0b8a82c03133bf2cbedf7bcdfc7947fd7932bc472eb WatchSource:0}: Error finding container 6d15ba4503b213e32bcaf0b8a82c03133bf2cbedf7bcdfc7947fd7932bc472eb: Status 404 returned error can't find the container with id 6d15ba4503b213e32bcaf0b8a82c03133bf2cbedf7bcdfc7947fd7932bc472eb
Oct 31 03:28:19.657149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.657103    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:19.657263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.657221    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:19.657342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:19.657271    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:20.209633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.209360    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg" event={"ID":"b542a9bd-4c75-437f-a939-559b47450e6a","Type":"ContainerDied","Data":"5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc"}
Oct 31 03:28:20.209633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.209428    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5bacec4b61f3163e23627df2aba645fae2a62b2cc25f3b8585f2755a90ebbc"
Oct 31 03:28:20.209633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.209499    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"
Oct 31 03:28:20.213588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.213485    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" event={"ID":"9154c52d-f1a1-45a0-803a-c3384b81c35d","Type":"ContainerStarted","Data":"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"}
Oct 31 03:28:20.213588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.213546    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" event={"ID":"9154c52d-f1a1-45a0-803a-c3384b81c35d","Type":"ContainerStarted","Data":"6d15ba4503b213e32bcaf0b8a82c03133bf2cbedf7bcdfc7947fd7932bc472eb"}
Oct 31 03:28:20.242383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.242321    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podStartSLOduration=2.242303357 podStartE2EDuration="2.242303357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:20.236988609 +0000 UTC m=+774.762208198" watchObservedRunningTime="2026-10-31 03:28:20.242303357 +0000 UTC m=+774.767522945"
Oct 31 03:28:20.340672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.340293    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:20.341715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.341689    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:20.341843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.341735    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:20.461921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.461715    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:20.637648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.636736    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\" (UniqueName: \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\") pod \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") "
Oct 31 03:28:20.637648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.636923    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\") pod \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") "
Oct 31 03:28:20.637648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.637068    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\") pod \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\" (UID: \"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63\") "
Oct 31 03:28:20.638502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.638470    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca" pod "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" (UID: "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:20.641480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.641439    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn" pod "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" (UID: "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63"). InnerVolumeSpecName "kube-api-access-nddfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:20.650168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.650126    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca" pod "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" (UID: "1c4b59b8-d383-459b-87ad-fa8eaf0cbe63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:20.740349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.739580    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:20.740349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.739639    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:20.740349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:20.739659    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nddfn\" (UniqueName: \"kubernetes.io/projected/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63-kube-api-access-nddfn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:21.224659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:21.224591    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"
Oct 31 03:28:21.225036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:21.224731    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n" event={"ID":"1c4b59b8-d383-459b-87ad-fa8eaf0cbe63","Type":"ContainerDied","Data":"570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562"}
Oct 31 03:28:21.225036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:21.224765    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="570d76cdb177e1ed4b46c552d175bd215b7633a6eab2056c2b9a602343079562"
Oct 31 03:28:21.339923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:21.339884    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:21.340106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:21.339950    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:22.339944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:22.339908    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:22.340424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:22.339981    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:23.339958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.339915    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:23.340356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.339984    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:23.578670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.578571    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"]
Oct 31 03:28:23.579747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579719    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" containerName="revoke-keys"
Oct 31 03:28:23.579747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579746    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" containerName="revoke-keys"
Oct 31 03:28:23.579939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579776    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b542a9bd-4c75-437f-a939-559b47450e6a" containerName="revoke-keys"
Oct 31 03:28:23.579939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579792    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b542a9bd-4c75-437f-a939-559b47450e6a" containerName="revoke-keys"
Oct 31 03:28:23.579939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579937    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" containerName="revoke-keys"
Oct 31 03:28:23.580091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.579952    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b542a9bd-4c75-437f-a939-559b47450e6a" containerName="revoke-keys"
Oct 31 03:28:23.584279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.584255    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"]
Oct 31 03:28:23.584431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.584369    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.591697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.590508    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:23.591697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.590523    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:23.591697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.590521    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:23.591697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.590637    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:28:23.598389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.598356    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xw6g\" (UniqueName: \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.598565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.598543    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.599185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.598625    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.699877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.699839    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.700054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.699892    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.700054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.699973    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xw6g\" (UniqueName: \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.700530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.700506    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.702534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.702508    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.709707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.709672    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xw6g\" (UniqueName: \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\") pod \"maas-api-revoke-sub-c3cae3c3-qvn9b\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:23.903717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:23.903632    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:24.223749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.223711    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"]
Oct 31 03:28:24.224006 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:24.223982    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896d7527_15a3_4351_b04d_6d0cc8abfe0f.slice/crio-abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92 WatchSource:0}: Error finding container abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92: Status 404 returned error can't find the container with id abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92
Oct 31 03:28:24.247669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.247640    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b" event={"ID":"896d7527-15a3-4351-b04d-6d0cc8abfe0f","Type":"ContainerStarted","Data":"abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92"}
Oct 31 03:28:24.340740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.340688    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:24.341229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.340740    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:24.424391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.424341    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:24.439010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:24.438982    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:25.281861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:25.281808    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b" event={"ID":"896d7527-15a3-4351-b04d-6d0cc8abfe0f","Type":"ContainerStarted","Data":"2d25dd4a68debd4891a12aea0184a4e894ddc8337c72fe2c6aeb8809320bd908"}
Oct 31 03:28:25.294418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:25.294391    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:28:25.302050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:25.301996    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b" podStartSLOduration=2.301981466 podStartE2EDuration="2.301981466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:25.299531896 +0000 UTC m=+779.824751488" watchObservedRunningTime="2026-10-31 03:28:25.301981466 +0000 UTC m=+779.827201054"
Oct 31 03:28:25.344188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:25.344053    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:25.344188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:25.344123    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:26.076629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.076566    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"]
Oct 31 03:28:26.082167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.082138    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"]
Oct 31 03:28:26.082346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.082236    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.134733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.134700    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.134889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.134744    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.134889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.134833    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6q9w\" (UniqueName: \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.235931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.235894    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.236170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.235940    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.236170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.235972    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6q9w\" (UniqueName: \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.236584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.236560    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.238405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.238379    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.246363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.246333    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6q9w\" (UniqueName: \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\") pod \"maas-api-revoke-sub-b82a4524-4zm57\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.291956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.291918    5098 generic.go:356] "Generic (PLEG): container finished" podID="896d7527-15a3-4351-b04d-6d0cc8abfe0f" containerID="2d25dd4a68debd4891a12aea0184a4e894ddc8337c72fe2c6aeb8809320bd908" exitCode=0
Oct 31 03:28:26.292143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.292011    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b" event={"ID":"896d7527-15a3-4351-b04d-6d0cc8abfe0f","Type":"ContainerDied","Data":"2d25dd4a68debd4891a12aea0184a4e894ddc8337c72fe2c6aeb8809320bd908"}
Oct 31 03:28:26.341691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.339967    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 03:28:26.341691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.340026    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 03:28:26.399096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.399061    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:26.685209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:26.685170    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"]
Oct 31 03:28:26.685707 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:26.685672    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcadd9b3d_46de_4cc6_a180_01adb0b0156b.slice/crio-5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438 WatchSource:0}: Error finding container 5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438: Status 404 returned error can't find the container with id 5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438
Oct 31 03:28:27.300849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.300810    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57" event={"ID":"cadd9b3d-46de-4cc6-a180-01adb0b0156b","Type":"ContainerStarted","Data":"58f476553f54f11ee0e5387b1701f7c8b70e9557112994e534fbc0b12f7a2537"}
Oct 31 03:28:27.301034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.300867    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57" event={"ID":"cadd9b3d-46de-4cc6-a180-01adb0b0156b","Type":"ContainerStarted","Data":"5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438"}
Oct 31 03:28:27.321803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.321687    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57" podStartSLOduration=1.321671499 podStartE2EDuration="1.321671499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:27.318350246 +0000 UTC m=+781.843569836" watchObservedRunningTime="2026-10-31 03:28:27.321671499 +0000 UTC m=+781.846891088"
Oct 31 03:28:27.347220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.347184    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:27.347858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.347792    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:27.350373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.349762    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:28:27.350846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.350824    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"]
Oct 31 03:28:27.355628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.355328    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.368653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.362636    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"]
Oct 31 03:28:27.459654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.457897    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.459654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.458135    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.459654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.458776    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xbn\" (UniqueName: \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.519920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.519887    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:27.559343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.559305    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\") pod \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") "
Oct 31 03:28:27.559597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.559354    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\") pod \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") "
Oct 31 03:28:27.559718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.559630    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\" (UniqueName: \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\") pod \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\" (UID: \"896d7527-15a3-4351-b04d-6d0cc8abfe0f\") "
Oct 31 03:28:27.559949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.559917    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.560191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.560037    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca" pod "896d7527-15a3-4351-b04d-6d0cc8abfe0f" (UID: "896d7527-15a3-4351-b04d-6d0cc8abfe0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:27.560191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.560044    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xbn\" (UniqueName: \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.560191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.560143    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.560336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.560251    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:27.561175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.561152    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.562979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.562951    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g" pod "896d7527-15a3-4351-b04d-6d0cc8abfe0f" (UID: "896d7527-15a3-4351-b04d-6d0cc8abfe0f"). InnerVolumeSpecName "kube-api-access-2xw6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:27.564214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.564190    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.565729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.565706    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca" pod "896d7527-15a3-4351-b04d-6d0cc8abfe0f" (UID: "896d7527-15a3-4351-b04d-6d0cc8abfe0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:27.568696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.568582    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xbn\" (UniqueName: \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\") pod \"maas-api-revoke-sub-9ca791c1-74bl2\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.661521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.661486    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xw6g\" (UniqueName: \"kubernetes.io/projected/896d7527-15a3-4351-b04d-6d0cc8abfe0f-kube-api-access-2xw6g\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:27.661521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.661519    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/896d7527-15a3-4351-b04d-6d0cc8abfe0f-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:27.681059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.681020    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:27.962000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:27.961958    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"]
Oct 31 03:28:27.962165 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:27.962016    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac0fdad_1bd5_4d63_a1e3_61df3a9005f6.slice/crio-d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca WatchSource:0}: Error finding container d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca: Status 404 returned error can't find the container with id d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca
Oct 31 03:28:28.310049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.310021    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"
Oct 31 03:28:28.310266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.310020    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b" event={"ID":"896d7527-15a3-4351-b04d-6d0cc8abfe0f","Type":"ContainerDied","Data":"abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92"}
Oct 31 03:28:28.310266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.310141    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abda7035e225d031192ccbe177d124042e7fae3be95913ee9ecaff7ff1d86e92"
Oct 31 03:28:28.312572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.312529    5098 generic.go:356] "Generic (PLEG): container finished" podID="cadd9b3d-46de-4cc6-a180-01adb0b0156b" containerID="58f476553f54f11ee0e5387b1701f7c8b70e9557112994e534fbc0b12f7a2537" exitCode=0
Oct 31 03:28:28.312817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.312767    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57" event={"ID":"cadd9b3d-46de-4cc6-a180-01adb0b0156b","Type":"ContainerDied","Data":"58f476553f54f11ee0e5387b1701f7c8b70e9557112994e534fbc0b12f7a2537"}
Oct 31 03:28:28.314657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.314630    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2" event={"ID":"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6","Type":"ContainerStarted","Data":"85ca7dc02a414e019b512a03335bc91be45144a58bb7902882c67d730a006898"}
Oct 31 03:28:28.314766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.314681    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2" event={"ID":"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6","Type":"ContainerStarted","Data":"d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca"}
Oct 31 03:28:28.353936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:28.353880    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2" podStartSLOduration=1.353866937 podStartE2EDuration="1.353866937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:28.351730479 +0000 UTC m=+782.876950068" watchObservedRunningTime="2026-10-31 03:28:28.353866937 +0000 UTC m=+782.879086558"
Oct 31 03:28:29.322777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.322737    5098 generic.go:356] "Generic (PLEG): container finished" podID="6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" containerID="85ca7dc02a414e019b512a03335bc91be45144a58bb7902882c67d730a006898" exitCode=0
Oct 31 03:28:29.323263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.322825    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2" event={"ID":"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6","Type":"ContainerDied","Data":"85ca7dc02a414e019b512a03335bc91be45144a58bb7902882c67d730a006898"}
Oct 31 03:28:29.501723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.501695    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:29.584649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.584550    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\") pod \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") "
Oct 31 03:28:29.584803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.584708    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\") pod \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") "
Oct 31 03:28:29.584803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.584744    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\" (UniqueName: \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\") pod \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\" (UID: \"cadd9b3d-46de-4cc6-a180-01adb0b0156b\") "
Oct 31 03:28:29.585120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.585090    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca" pod "cadd9b3d-46de-4cc6-a180-01adb0b0156b" (UID: "cadd9b3d-46de-4cc6-a180-01adb0b0156b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:29.585213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.585178    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:29.586678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.586648    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca" pod "cadd9b3d-46de-4cc6-a180-01adb0b0156b" (UID: "cadd9b3d-46de-4cc6-a180-01adb0b0156b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:29.586799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.586782    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w" pod "cadd9b3d-46de-4cc6-a180-01adb0b0156b" (UID: "cadd9b3d-46de-4cc6-a180-01adb0b0156b"). InnerVolumeSpecName "kube-api-access-z6q9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:29.686296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.686256    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6q9w\" (UniqueName: \"kubernetes.io/projected/cadd9b3d-46de-4cc6-a180-01adb0b0156b-kube-api-access-z6q9w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:29.686296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:29.686288    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cadd9b3d-46de-4cc6-a180-01adb0b0156b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:30.332063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.332027    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"
Oct 31 03:28:30.332548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.332061    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57" event={"ID":"cadd9b3d-46de-4cc6-a180-01adb0b0156b","Type":"ContainerDied","Data":"5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438"}
Oct 31 03:28:30.332548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.332127    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b317168dc058dd25f0b10758a0683471434505160ce99df89f9c0babcf30438"
Oct 31 03:28:30.522082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.522054    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:30.599947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.599861    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\" (UniqueName: \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\") pod \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") "
Oct 31 03:28:30.600086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.599971    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\") pod \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") "
Oct 31 03:28:30.600086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.600011    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\") pod \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\" (UID: \"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6\") "
Oct 31 03:28:30.600365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.600332    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca" pod "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" (UID: "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:30.602229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.602203    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn" pod "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" (UID: "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6"). InnerVolumeSpecName "kube-api-access-h9xbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:30.602333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.602251    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca" pod "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" (UID: "6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:30.702030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.701991    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:30.702030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.702027    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:30.702030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:30.702038    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xbn\" (UniqueName: \"kubernetes.io/projected/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6-kube-api-access-h9xbn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:31.059366 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:28:31.059326    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:28:31.341374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.341270    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2" event={"ID":"6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6","Type":"ContainerDied","Data":"d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca"}
Oct 31 03:28:31.341374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.341334    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d22d91874db571ac0160bbdf93c7803680551f2dbfbe00ba926f264841a879ca"
Oct 31 03:28:31.341374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.341292    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"
Oct 31 03:28:31.390753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.390715    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"]
Oct 31 03:28:31.391288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391273    5098 cpu_manager.go:414] "containerMap: removing container" podUID="896d7527-15a3-4351-b04d-6d0cc8abfe0f" containerName="revoke-keys"
Oct 31 03:28:31.391343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391289    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="896d7527-15a3-4351-b04d-6d0cc8abfe0f" containerName="revoke-keys"
Oct 31 03:28:31.391343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391315    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cadd9b3d-46de-4cc6-a180-01adb0b0156b" containerName="revoke-keys"
Oct 31 03:28:31.391343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391320    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cadd9b3d-46de-4cc6-a180-01adb0b0156b" containerName="revoke-keys"
Oct 31 03:28:31.391429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391348    5098 cpu_manager.go:414] "containerMap: removing container" podUID="6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" containerName="revoke-keys"
Oct 31 03:28:31.391429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391353    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" containerName="revoke-keys"
Oct 31 03:28:31.391486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391441    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cadd9b3d-46de-4cc6-a180-01adb0b0156b" containerName="revoke-keys"
Oct 31 03:28:31.391486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391452    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" containerName="revoke-keys"
Oct 31 03:28:31.391486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.391461    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="896d7527-15a3-4351-b04d-6d0cc8abfe0f" containerName="revoke-keys"
Oct 31 03:28:31.395147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.395126    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.400787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.400763    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:31.400926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.400763    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:31.401725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.401674    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:31.401858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.401748    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:28:31.402552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.402529    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"]
Oct 31 03:28:31.513004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.512963    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l858s\" (UniqueName: \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.513193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.513098    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.513193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.513142    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.545818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.545783    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:28:31.614244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.614158    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l858s\" (UniqueName: \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.614244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.614240    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.614463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.614274    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.615015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.614986    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.616833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.616807    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.623472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.623441    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l858s\" (UniqueName: \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\") pod \"maas-api-revoke-sub-720fc28c-xc6cd\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:31.715509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:31.715468    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:32.022351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:32.022319    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"]
Oct 31 03:28:32.022555 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:32.022498    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b71ba38_3041_4c42_8529_bddde79040e3.slice/crio-67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592 WatchSource:0}: Error finding container 67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592: Status 404 returned error can't find the container with id 67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592
Oct 31 03:28:32.370879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:32.370267    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd" event={"ID":"5b71ba38-3041-4c42-8529-bddde79040e3","Type":"ContainerStarted","Data":"67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592"}
Oct 31 03:28:33.405689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.405227    5098 generic.go:356] "Generic (PLEG): container finished" podID="5b71ba38-3041-4c42-8529-bddde79040e3" containerID="8a4fd03dbad5678098671c355d7c435501b5ab879157fed2c28b34ea12e92ad3" exitCode=0
Oct 31 03:28:33.405689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.405505    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd" event={"ID":"5b71ba38-3041-4c42-8529-bddde79040e3","Type":"ContainerDied","Data":"8a4fd03dbad5678098671c355d7c435501b5ab879157fed2c28b34ea12e92ad3"}
Oct 31 03:28:33.687834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.687693    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:28:33.726012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.724480    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:28:33.726012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.724619    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.851264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.851225    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"]
Oct 31 03:28:33.856575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.856545    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:33.888420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.888384    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"]
Oct 31 03:28:33.896397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896355    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896423    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896520    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896647    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896706    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896731    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896795    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896825    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.896965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.896875    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmqt\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.998690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998558    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.998690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998675    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:33.998952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998697    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smqrt\" (UniqueName: \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:33.998952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998750    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.998952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998922    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998952    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.998996    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999016    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:33.999111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999048    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999119    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmqt\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999154    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999193    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999302    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999412    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999518    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999586    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:33.999768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:33.999657    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.001199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.001173    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.001468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.001450    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.009955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.009925    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.010114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.010046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmqt\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.042356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.042324    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:34.102304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.102261    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smqrt\" (UniqueName: \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.102465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.102445    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.102682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.102663    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.103142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.103119    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.105166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.105140    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.119074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.119040    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smqrt\" (UniqueName: \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\") pod \"maas-api-revoke-sub-9c7f304f-g82dx\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.199896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.199859    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:28:34.200096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.199979    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:34.214392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.207986    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.233414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.233384    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:28:34.305569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305531    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305585    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305688    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305724    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305767    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305800    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305866    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305903    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.305936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.305921    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.332907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.332867    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:28:34.336359 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:34.336312    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0cb7e52_bfbf_4fb0_a5d6_ad9a17f89edf.slice/crio-06f4b96df038ee3420b716632d30f83b394ce047da3b82e26b2da7b4cf6ff6e9 WatchSource:0}: Error finding container 06f4b96df038ee3420b716632d30f83b394ce047da3b82e26b2da7b4cf6ff6e9: Status 404 returned error can't find the container with id 06f4b96df038ee3420b716632d30f83b394ce047da3b82e26b2da7b4cf6ff6e9
Oct 31 03:28:34.339221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.339185    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:34.339331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.339257    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:34.339331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.339291    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:34.406995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.406966    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407004    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407033    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407070    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407131    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407181    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407287    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407342    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407375    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407416    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407650    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407687    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407740    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.408150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.407877    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.410336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.410311    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.410440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.410409    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.415662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.415635    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" event={"ID":"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf","Type":"ContainerStarted","Data":"06f4b96df038ee3420b716632d30f83b394ce047da3b82e26b2da7b4cf6ff6e9"}
Oct 31 03:28:34.415967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.415948    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.416047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.416027    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.518993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.518551    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"]
Oct 31 03:28:34.550429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.550400    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:34.576512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.576479    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:34.714347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.714316    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\") pod \"5b71ba38-3041-4c42-8529-bddde79040e3\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") "
Oct 31 03:28:34.714565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.714457    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\" (UniqueName: \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\") pod \"5b71ba38-3041-4c42-8529-bddde79040e3\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") "
Oct 31 03:28:34.714565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.714527    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\") pod \"5b71ba38-3041-4c42-8529-bddde79040e3\" (UID: \"5b71ba38-3041-4c42-8529-bddde79040e3\") "
Oct 31 03:28:34.714909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.714878    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca" pod "5b71ba38-3041-4c42-8529-bddde79040e3" (UID: "5b71ba38-3041-4c42-8529-bddde79040e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:34.715089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.715072    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.750034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.749989    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca" pod "5b71ba38-3041-4c42-8529-bddde79040e3" (UID: "5b71ba38-3041-4c42-8529-bddde79040e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:34.750034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.749993    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s" pod "5b71ba38-3041-4c42-8529-bddde79040e3" (UID: "5b71ba38-3041-4c42-8529-bddde79040e3"). InnerVolumeSpecName "kube-api-access-l858s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:34.816141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.815993    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5b71ba38-3041-4c42-8529-bddde79040e3-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.816141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.816032    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l858s\" (UniqueName: \"kubernetes.io/projected/5b71ba38-3041-4c42-8529-bddde79040e3-kube-api-access-l858s\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:34.849673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.849636    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:28:34.850516 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:34.850487    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b42b50_0b86_484f_9e70_f56ae576c4d0.slice/crio-78b8e000cd0dd150c4039ffa4c22169b220b666198cc0db20b34545cbb8d9dcc WatchSource:0}: Error finding container 78b8e000cd0dd150c4039ffa4c22169b220b666198cc0db20b34545cbb8d9dcc: Status 404 returned error can't find the container with id 78b8e000cd0dd150c4039ffa4c22169b220b666198cc0db20b34545cbb8d9dcc
Oct 31 03:28:34.853558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.853526    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:34.853679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.853592    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:34.853679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:34.853639    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:35.424769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.424717    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx" event={"ID":"f6a68d99-3008-4468-99de-d799fb336c02","Type":"ContainerStarted","Data":"8431f881b7650c0a6f851bad0933024a154d24aa47365a08da83637241f0e1e8"}
Oct 31 03:28:35.425235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.424817    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx" event={"ID":"f6a68d99-3008-4468-99de-d799fb336c02","Type":"ContainerStarted","Data":"757460037ce3cf405001b1105311e577709c96443515547f902622dab4b4f7fc"}
Oct 31 03:28:35.426197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.426173    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"
Oct 31 03:28:35.426349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.426171    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd" event={"ID":"5b71ba38-3041-4c42-8529-bddde79040e3","Type":"ContainerDied","Data":"67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592"}
Oct 31 03:28:35.426349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.426289    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67702ae283f7723d5a99cde3caed82533910b65b7a7c87d6292acacc986ca592"
Oct 31 03:28:35.427958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.427931    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" event={"ID":"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf","Type":"ContainerStarted","Data":"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"}
Oct 31 03:28:35.429986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.429956    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" event={"ID":"b6b42b50-0b86-484f-9e70-f56ae576c4d0","Type":"ContainerStarted","Data":"8e18f430341a50f6c639e5c3338507e39cea999cbd2481918611114d5fbfa740"}
Oct 31 03:28:35.430140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.430007    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" event={"ID":"b6b42b50-0b86-484f-9e70-f56ae576c4d0","Type":"ContainerStarted","Data":"78b8e000cd0dd150c4039ffa4c22169b220b666198cc0db20b34545cbb8d9dcc"}
Oct 31 03:28:35.481632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.475340    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx" podStartSLOduration=2.475316996 podStartE2EDuration="2.475316996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:35.443794777 +0000 UTC m=+789.969014357" watchObservedRunningTime="2026-10-31 03:28:35.475316996 +0000 UTC m=+790.000536588"
Oct 31 03:28:35.515064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.515009    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podStartSLOduration=1.5149948960000001 podStartE2EDuration="1.514994896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:35.483315316 +0000 UTC m=+790.008534919" watchObservedRunningTime="2026-10-31 03:28:35.514994896 +0000 UTC m=+790.040214485"
Oct 31 03:28:35.515266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.515245    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podStartSLOduration=2.515240058 podStartE2EDuration="2.515240058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:35.513286546 +0000 UTC m=+790.038506131" watchObservedRunningTime="2026-10-31 03:28:35.515240058 +0000 UTC m=+790.040459647"
Oct 31 03:28:35.557045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.557009    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:35.576851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.572006    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:35.576851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:35.572084    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:36.042826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.042784    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:36.044777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.044748    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:36.044897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.044805    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:36.307759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.307668    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"]
Oct 31 03:28:36.308933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.308901    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5b71ba38-3041-4c42-8529-bddde79040e3" containerName="revoke-keys"
Oct 31 03:28:36.309109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.309094    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b71ba38-3041-4c42-8529-bddde79040e3" containerName="revoke-keys"
Oct 31 03:28:36.309431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.309380    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b71ba38-3041-4c42-8529-bddde79040e3" containerName="revoke-keys"
Oct 31 03:28:36.313139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.313121    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"]
Oct 31 03:28:36.313295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.313266    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.440074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.440041    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.440524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.440252    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhwt\" (UniqueName: \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.440524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.440312    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.440673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.440617    5098 generic.go:356] "Generic (PLEG): container finished" podID="f6a68d99-3008-4468-99de-d799fb336c02" containerID="8431f881b7650c0a6f851bad0933024a154d24aa47365a08da83637241f0e1e8" exitCode=0
Oct 31 03:28:36.440730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.440699    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx" event={"ID":"f6a68d99-3008-4468-99de-d799fb336c02","Type":"ContainerDied","Data":"8431f881b7650c0a6f851bad0933024a154d24aa47365a08da83637241f0e1e8"}
Oct 31 03:28:36.541314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.541283    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.542095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.542075    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhwt\" (UniqueName: \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.542739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.542717    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.544146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.544120    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.565744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.560078    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:36.566069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.566028    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:36.575643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.570077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.604142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.604109    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhwt\" (UniqueName: \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\") pod \"maas-api-revoke-sub-996c90ee-jb6rx\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:36.629816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:36.629785    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:37.015973 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:37.015943    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a0e5ba1_4695_48de_9607_3a668b2beff0.slice/crio-8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d WatchSource:0}: Error finding container 8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d: Status 404 returned error can't find the container with id 8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d
Oct 31 03:28:37.016109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.016008    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"]
Oct 31 03:28:37.043857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.043505    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:37.043857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.043550    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:37.493769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.484901    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx" event={"ID":"6a0e5ba1-4695-48de-9607-3a668b2beff0","Type":"ContainerStarted","Data":"8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d"}
Oct 31 03:28:37.609142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.608971    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:37.609142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.609030    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:37.912841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:37.912804    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:38.026064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.025965    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\") pod \"f6a68d99-3008-4468-99de-d799fb336c02\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") "
Oct 31 03:28:38.026064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.026035    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\" (UniqueName: \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\") pod \"f6a68d99-3008-4468-99de-d799fb336c02\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") "
Oct 31 03:28:38.026312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.026172    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\") pod \"f6a68d99-3008-4468-99de-d799fb336c02\" (UID: \"f6a68d99-3008-4468-99de-d799fb336c02\") "
Oct 31 03:28:38.027085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.027001    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca" pod "f6a68d99-3008-4468-99de-d799fb336c02" (UID: "f6a68d99-3008-4468-99de-d799fb336c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:38.031869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.031824    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt" pod "f6a68d99-3008-4468-99de-d799fb336c02" (UID: "f6a68d99-3008-4468-99de-d799fb336c02"). InnerVolumeSpecName "kube-api-access-smqrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:38.032302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.032272    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca" pod "f6a68d99-3008-4468-99de-d799fb336c02" (UID: "f6a68d99-3008-4468-99de-d799fb336c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:38.043726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.043689    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:38.044207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.043751    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:38.128686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.128489    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:38.128686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.128533    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6a68d99-3008-4468-99de-d799fb336c02-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:38.128686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.128549    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smqrt\" (UniqueName: \"kubernetes.io/projected/f6a68d99-3008-4468-99de-d799fb336c02-kube-api-access-smqrt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:38.507742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.507648    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx" event={"ID":"f6a68d99-3008-4468-99de-d799fb336c02","Type":"ContainerDied","Data":"757460037ce3cf405001b1105311e577709c96443515547f902622dab4b4f7fc"}
Oct 31 03:28:38.507742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.507727    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="757460037ce3cf405001b1105311e577709c96443515547f902622dab4b4f7fc"
Oct 31 03:28:38.508311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.507781    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"
Oct 31 03:28:38.511165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.511132    5098 generic.go:356] "Generic (PLEG): container finished" podID="6a0e5ba1-4695-48de-9607-3a668b2beff0" containerID="3cf3c18a7e98c3d210657262ce93ac520c7f9de646d5530bdc9f81a76d21004b" exitCode=0
Oct 31 03:28:38.511352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.511227    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx" event={"ID":"6a0e5ba1-4695-48de-9607-3a668b2beff0","Type":"ContainerDied","Data":"3cf3c18a7e98c3d210657262ce93ac520c7f9de646d5530bdc9f81a76d21004b"}
Oct 31 03:28:38.579819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.579772    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:38.580123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:38.580091    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:39.043697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.043663    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:39.043886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.043717    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:39.573166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.573128    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:39.575023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.573738    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:39.823920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.823887    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:39.972631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.972130    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\") pod \"6a0e5ba1-4695-48de-9607-3a668b2beff0\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") "
Oct 31 03:28:39.972631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.972203    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\" (UniqueName: \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\") pod \"6a0e5ba1-4695-48de-9607-3a668b2beff0\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") "
Oct 31 03:28:39.972631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.972273    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\") pod \"6a0e5ba1-4695-48de-9607-3a668b2beff0\" (UID: \"6a0e5ba1-4695-48de-9607-3a668b2beff0\") "
Oct 31 03:28:39.998931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:39.998881    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca" pod "6a0e5ba1-4695-48de-9607-3a668b2beff0" (UID: "6a0e5ba1-4695-48de-9607-3a668b2beff0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:40.001057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.001019    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt" pod "6a0e5ba1-4695-48de-9607-3a668b2beff0" (UID: "6a0e5ba1-4695-48de-9607-3a668b2beff0"). InnerVolumeSpecName "kube-api-access-hlhwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:40.001963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.001934    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca" pod "6a0e5ba1-4695-48de-9607-3a668b2beff0" (UID: "6a0e5ba1-4695-48de-9607-3a668b2beff0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:40.042943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.042913    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:40.043141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.042969    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:40.076688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.075518    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:40.076688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.075568    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlhwt\" (UniqueName: \"kubernetes.io/projected/6a0e5ba1-4695-48de-9607-3a668b2beff0-kube-api-access-hlhwt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:40.076688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.075587    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a0e5ba1-4695-48de-9607-3a668b2beff0-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:40.527781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.527752    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"
Oct 31 03:28:40.527931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.527802    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx" event={"ID":"6a0e5ba1-4695-48de-9607-3a668b2beff0","Type":"ContainerDied","Data":"8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d"}
Oct 31 03:28:40.527931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.527852    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8aa691dc38308f1299e793906a0f8de8ba7f238a8c0b54b59e38274801ade52d"
Oct 31 03:28:40.562147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.562112    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:40.562436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:40.562404    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:41.043837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.043798    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 31 03:28:41.044276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.043869    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 31 03:28:41.502323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.502225    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"]
Oct 31 03:28:41.503193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503161    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f6a68d99-3008-4468-99de-d799fb336c02" containerName="revoke-keys"
Oct 31 03:28:41.503193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503187    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a68d99-3008-4468-99de-d799fb336c02" containerName="revoke-keys"
Oct 31 03:28:41.503398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503224    5098 cpu_manager.go:414] "containerMap: removing container" podUID="6a0e5ba1-4695-48de-9607-3a668b2beff0" containerName="revoke-keys"
Oct 31 03:28:41.503398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503233    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a0e5ba1-4695-48de-9607-3a668b2beff0" containerName="revoke-keys"
Oct 31 03:28:41.503398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503383    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a68d99-3008-4468-99de-d799fb336c02" containerName="revoke-keys"
Oct 31 03:28:41.503596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.503405    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a0e5ba1-4695-48de-9607-3a668b2beff0" containerName="revoke-keys"
Oct 31 03:28:41.507856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.507832    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.522196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.522164    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:28:41.522196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.522184    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:28:41.522196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.522191    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:28:41.522481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.522461    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:28:41.533599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.533552    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"]
Oct 31 03:28:41.558413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.558377    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body=
Oct 31 03:28:41.558829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.558784    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused"
Oct 31 03:28:41.622847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.622810    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.623022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.622955    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zbb4\" (UniqueName: \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.623022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.622993    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.724259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.724215    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.724445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.724337    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zbb4\" (UniqueName: \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.724445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.724376    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.725116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.725088    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.726875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.726852    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.738332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.738305    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zbb4\" (UniqueName: \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\") pod \"maas-api-revoke-sub-692ce641-fd4lp\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:41.827473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:41.827431    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:42.061076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.061042    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:42.061591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.061574    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:42.062718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.062698    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:28:42.139860 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:42.139801    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b835641_630f_4812_9e40_9b0afe0756bb.slice/crio-58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f WatchSource:0}: Error finding container 58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f: Status 404 returned error can't find the container with id 58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f
Oct 31 03:28:42.140144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.140085    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"]
Oct 31 03:28:42.556576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.556544    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:42.557086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.557058    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:42.557975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.557955    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:28:42.560620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.560584    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp" event={"ID":"0b835641-630f-4812-9e40-9b0afe0756bb","Type":"ContainerStarted","Data":"bd7a4c3226757cefc2be9a5100df05fb96cb407cdaf2e650f19e507330d49ce3"}
Oct 31 03:28:42.560730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.560642    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp" event={"ID":"0b835641-630f-4812-9e40-9b0afe0756bb","Type":"ContainerStarted","Data":"58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f"}
Oct 31 03:28:42.653768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:42.653701    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp" podStartSLOduration=1.653678358 podStartE2EDuration="1.653678358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:42.649503018 +0000 UTC m=+797.174722608" watchObservedRunningTime="2026-10-31 03:28:42.653678358 +0000 UTC m=+797.178897964"
Oct 31 03:28:43.250307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.250229    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"]
Oct 31 03:28:43.254806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.254784    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.262971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.262942    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"]
Oct 31 03:28:43.269587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.269549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsw6l\" (UniqueName: \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.269755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.269666    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.269816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.269799    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.370681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.370643    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsw6l\" (UniqueName: \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.370859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.370694    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.370859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.370741    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.371462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.371440    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.373220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.373195    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.379244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.379219    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsw6l\" (UniqueName: \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\") pod \"maas-api-revoke-sub-0f4d101f-fsh9k\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.569751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.569716    5098 generic.go:356] "Generic (PLEG): container finished" podID="0b835641-630f-4812-9e40-9b0afe0756bb" containerID="bd7a4c3226757cefc2be9a5100df05fb96cb407cdaf2e650f19e507330d49ce3" exitCode=0
Oct 31 03:28:43.569948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.569799    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp" event={"ID":"0b835641-630f-4812-9e40-9b0afe0756bb","Type":"ContainerDied","Data":"bd7a4c3226757cefc2be9a5100df05fb96cb407cdaf2e650f19e507330d49ce3"}
Oct 31 03:28:43.570591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.570570    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:43.875082 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:43.875040    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb94fba14_6580_4dab_97b5_b3e2295ee918.slice/crio-1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662 WatchSource:0}: Error finding container 1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662: Status 404 returned error can't find the container with id 1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662
Oct 31 03:28:43.875082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:43.875050    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"]
Oct 31 03:28:44.583945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.583459    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k" event={"ID":"b94fba14-6580-4dab-97b5-b3e2295ee918","Type":"ContainerStarted","Data":"ac90a6cec8ab24231fd8ae0280b2578b27404a875ee4bccbb3fc8062083edf20"}
Oct 31 03:28:44.583945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.583530    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k" event={"ID":"b94fba14-6580-4dab-97b5-b3e2295ee918","Type":"ContainerStarted","Data":"1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662"}
Oct 31 03:28:44.616865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.616798    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k" podStartSLOduration=1.616775541 podStartE2EDuration="1.616775541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:44.611130902 +0000 UTC m=+799.136350503" watchObservedRunningTime="2026-10-31 03:28:44.616775541 +0000 UTC m=+799.141995131"
Oct 31 03:28:44.827926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.827896    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:44.886300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.886203    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\" (UniqueName: \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\") pod \"0b835641-630f-4812-9e40-9b0afe0756bb\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") "
Oct 31 03:28:44.886470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.886382    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\") pod \"0b835641-630f-4812-9e40-9b0afe0756bb\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") "
Oct 31 03:28:44.886470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.886416    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\") pod \"0b835641-630f-4812-9e40-9b0afe0756bb\" (UID: \"0b835641-630f-4812-9e40-9b0afe0756bb\") "
Oct 31 03:28:44.886908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.886877    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca" pod "0b835641-630f-4812-9e40-9b0afe0756bb" (UID: "0b835641-630f-4812-9e40-9b0afe0756bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:44.887089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.886986    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:44.888750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.888712    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4" pod "0b835641-630f-4812-9e40-9b0afe0756bb" (UID: "0b835641-630f-4812-9e40-9b0afe0756bb"). InnerVolumeSpecName "kube-api-access-8zbb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:44.889301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.889237    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca" pod "0b835641-630f-4812-9e40-9b0afe0756bb" (UID: "0b835641-630f-4812-9e40-9b0afe0756bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:44.988457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.988415    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zbb4\" (UniqueName: \"kubernetes.io/projected/0b835641-630f-4812-9e40-9b0afe0756bb-kube-api-access-8zbb4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:44.988457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:44.988452    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b835641-630f-4812-9e40-9b0afe0756bb-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:45.591885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:45.591856    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"
Oct 31 03:28:45.592345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:45.591890    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp" event={"ID":"0b835641-630f-4812-9e40-9b0afe0756bb","Type":"ContainerDied","Data":"58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f"}
Oct 31 03:28:45.592345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:45.591940    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58a66231defea60cb5bc9f30dc0fd00cc16d5ef399b0bf55a4aa149e9244459f"
Oct 31 03:28:45.593656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:45.593630    5098 generic.go:356] "Generic (PLEG): container finished" podID="b94fba14-6580-4dab-97b5-b3e2295ee918" containerID="ac90a6cec8ab24231fd8ae0280b2578b27404a875ee4bccbb3fc8062083edf20" exitCode=0
Oct 31 03:28:45.593778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:45.593713    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k" event={"ID":"b94fba14-6580-4dab-97b5-b3e2295ee918","Type":"ContainerDied","Data":"ac90a6cec8ab24231fd8ae0280b2578b27404a875ee4bccbb3fc8062083edf20"}
Oct 31 03:28:46.771700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.771673    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:46.811261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.811226    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\") pod \"b94fba14-6580-4dab-97b5-b3e2295ee918\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") "
Oct 31 03:28:46.811454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.811391    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\") pod \"b94fba14-6580-4dab-97b5-b3e2295ee918\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") "
Oct 31 03:28:46.811522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.811484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\" (UniqueName: \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\") pod \"b94fba14-6580-4dab-97b5-b3e2295ee918\" (UID: \"b94fba14-6580-4dab-97b5-b3e2295ee918\") "
Oct 31 03:28:46.811870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.811834    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca" pod "b94fba14-6580-4dab-97b5-b3e2295ee918" (UID: "b94fba14-6580-4dab-97b5-b3e2295ee918"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:28:46.813910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.813878    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca" pod "b94fba14-6580-4dab-97b5-b3e2295ee918" (UID: "b94fba14-6580-4dab-97b5-b3e2295ee918"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:28:46.814017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.813900    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l" pod "b94fba14-6580-4dab-97b5-b3e2295ee918" (UID: "b94fba14-6580-4dab-97b5-b3e2295ee918"). InnerVolumeSpecName "kube-api-access-fsw6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:28:46.913377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.913289    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsw6l\" (UniqueName: \"kubernetes.io/projected/b94fba14-6580-4dab-97b5-b3e2295ee918-kube-api-access-fsw6l\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:46.913377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.913324    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:46.913377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:46.913335    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b94fba14-6580-4dab-97b5-b3e2295ee918-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:28:47.610085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:47.610053    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"
Oct 31 03:28:47.612619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:47.612582    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k" event={"ID":"b94fba14-6580-4dab-97b5-b3e2295ee918","Type":"ContainerDied","Data":"1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662"}
Oct 31 03:28:47.612769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:47.612639    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a61e5e7ef079a9aed627ec968b9f41582e28df02c96aeab9f4b1cf43f5e2662"
Oct 31 03:28:49.714688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.714645    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 03:28:49.715568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715543    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0b835641-630f-4812-9e40-9b0afe0756bb" containerName="revoke-keys"
Oct 31 03:28:49.715647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715573    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b835641-630f-4812-9e40-9b0afe0756bb" containerName="revoke-keys"
Oct 31 03:28:49.715647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715621    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b94fba14-6580-4dab-97b5-b3e2295ee918" containerName="revoke-keys"
Oct 31 03:28:49.715647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715632    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b94fba14-6580-4dab-97b5-b3e2295ee918" containerName="revoke-keys"
Oct 31 03:28:49.715868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715853    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b835641-630f-4812-9e40-9b0afe0756bb" containerName="revoke-keys"
Oct 31 03:28:49.715905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.715874    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b94fba14-6580-4dab-97b5-b3e2295ee918" containerName="revoke-keys"
Oct 31 03:28:49.721482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.721455    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.728410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.728383    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 03:28:49.731365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.731341    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7f3159-openshift-default-dockercfg-4mdls\""
Oct 31 03:28:49.751150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751107    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751201    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751231    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751318    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r82mm\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751348    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751402    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751474    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751560    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.751685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.751630    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.852598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.852551    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.852873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.852856    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.853043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.853029    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.854575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.854551    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.854869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.854850    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r82mm\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.854992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.854976    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.855126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.855112    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.855281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.855268    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.855442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.855430    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.855892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.855872    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.857484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.853900    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.857666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.856251    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.857785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.856478    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.857873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.856706    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.882492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.882454    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.920990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.920955    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.927716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.923147    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:49.942631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:49.936479    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r82mm\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:50.042272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.041960    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:50.433692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.432600    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 03:28:50.477095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.475661    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:50.477095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.475762    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:50.477095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.475819    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:28:50.693264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:50.693169    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" event={"ID":"c046f55b-2491-441e-9099-f9df90217489","Type":"ContainerStarted","Data":"c86055de9cbe0ea6d109cc45f32a793bf54c542cde37d2de5d9a2885366e8744"}
Oct 31 03:28:51.702732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:51.702690    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" event={"ID":"c046f55b-2491-441e-9099-f9df90217489","Type":"ContainerStarted","Data":"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"}
Oct 31 03:28:51.734623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:51.734553    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podStartSLOduration=2.7345331269999997 podStartE2EDuration="2.734533127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:51.729143145 +0000 UTC m=+806.254362737" watchObservedRunningTime="2026-10-31 03:28:51.734533127 +0000 UTC m=+806.259752718"
Oct 31 03:28:52.046632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:52.044736    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:52.046632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:52.044872    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:52.046632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:52.044917    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:53.043169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:53.043125    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:53.043569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:53.043194    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:54.042977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:54.042940    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:54.043141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:54.042994    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:55.043590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:55.043548    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:55.044027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:55.043646    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:56.043558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.043523    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:56.043771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.043579    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:56.702382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.702344    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 03:28:56.707115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.707087    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.714654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.714620    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 03:28:56.715157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.715131    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"kube-root-ca.crt\""
Oct 31 03:28:56.715318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.715160    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"default-dockercfg-bvtsj\""
Oct 31 03:28:56.716850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.716491    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"openshift-service-ca.crt\""
Oct 31 03:28:56.717022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.717000    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"distinct-w5-bcc7a4-kserve-self-signed-certs\""
Oct 31 03:28:56.818426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818385    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.818426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818427    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.818682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818464    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.818682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxhnj\" (UniqueName: \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.818748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818701    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.818780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.818762    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.921741    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.921854    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.922015    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.922044    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.922096    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.922124    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxhnj\" (UniqueName: \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.924922    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.925295    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.928211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.925530    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.931470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.930980    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.936745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.935508    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:56.937684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:56.937593    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxhnj\" (UniqueName: \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\") pod \"distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:57.023434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.023338    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:57.042966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.042933    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:57.043107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.043012    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:57.338807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.338776    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 03:28:57.371424 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:28:57.348919    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod510f30c0_9f70_4c49_b0c1_96fa9586aee8.slice/crio-7f5d24c2628b62230e168959cffb5fcdc5c5e1ff99f9cb3b5be34006b9f7b249 WatchSource:0}: Error finding container 7f5d24c2628b62230e168959cffb5fcdc5c5e1ff99f9cb3b5be34006b9f7b249: Status 404 returned error can't find the container with id 7f5d24c2628b62230e168959cffb5fcdc5c5e1ff99f9cb3b5be34006b9f7b249
Oct 31 03:28:57.754647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.754508    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" event={"ID":"510f30c0-9f70-4c49-b0c1-96fa9586aee8","Type":"ContainerStarted","Data":"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"}
Oct 31 03:28:57.754647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.754571    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" event={"ID":"510f30c0-9f70-4c49-b0c1-96fa9586aee8","Type":"ContainerStarted","Data":"7f5d24c2628b62230e168959cffb5fcdc5c5e1ff99f9cb3b5be34006b9f7b249"}
Oct 31 03:28:57.788638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:57.788558    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podStartSLOduration=1.788543061 podStartE2EDuration="1.788543061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:28:57.785387523 +0000 UTC m=+812.310607114" watchObservedRunningTime="2026-10-31 03:28:57.788543061 +0000 UTC m=+812.313762650"
Oct 31 03:28:58.023772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:58.023676    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:28:58.043432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:58.043392    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 03:28:58.043645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:58.043450    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 03:28:59.047464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:59.047425    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:59.047897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:59.047871    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:28:59.048679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:28:59.048647    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 03:29:01.058099 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:29:01.058061    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:29:03.358386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.358344    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:29:03.366077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.366044    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.370221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.370186    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:29:03.379142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.379109    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"openshift-service-ca.crt\""
Oct 31 03:29:03.379318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.379259    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"kube-root-ca.crt\""
Oct 31 03:29:03.379318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.379296    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"distinct-w1-6c6050-kserve-self-signed-certs\""
Oct 31 03:29:03.379435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.379340    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-6c6050\"/\"default-dockercfg-9w2c5\""
Oct 31 03:29:03.515004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.514967    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.515286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.515259    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.525826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.525780    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv9gf\" (UniqueName: \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.526035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.525936    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.526035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.525965    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.526035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.526008    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627065    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627146    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv9gf\" (UniqueName: \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627211    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627234    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627268    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627473    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627498    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627584    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.627664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.627637    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.629576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.629522    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.629782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.629763    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.635238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.635216    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv9gf\" (UniqueName: \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\") pod \"distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") " pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.690552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.690508    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:03.977233 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:03.977202    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2342cb51_d06e_44b6_bd12_bc80eedeed06.slice/crio-a47b4269872a53710d4badf763f8c25e3882832f708b7d2fb0a15778ef74fd60 WatchSource:0}: Error finding container a47b4269872a53710d4badf763f8c25e3882832f708b7d2fb0a15778ef74fd60: Status 404 returned error can't find the container with id a47b4269872a53710d4badf763f8c25e3882832f708b7d2fb0a15778ef74fd60
Oct 31 03:29:03.978202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:03.978162    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:29:04.811185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:04.811144    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" event={"ID":"2342cb51-d06e-44b6-bd12-bc80eedeed06","Type":"ContainerStarted","Data":"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"}
Oct 31 03:29:04.811699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:04.811194    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" event={"ID":"2342cb51-d06e-44b6-bd12-bc80eedeed06","Type":"ContainerStarted","Data":"a47b4269872a53710d4badf763f8c25e3882832f708b7d2fb0a15778ef74fd60"}
Oct 31 03:29:04.829939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:04.829883    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podStartSLOduration=1.829867112 podStartE2EDuration="1.829867112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:04.828895751 +0000 UTC m=+819.354115363" watchObservedRunningTime="2026-10-31 03:29:04.829867112 +0000 UTC m=+819.355086701"
Oct 31 03:29:05.691262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:05.691224    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:06.729888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.729851    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"]
Oct 31 03:29:06.734482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.734453    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.738434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.738405    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:06.739979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.739953    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:06.739979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.739961    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:06.740175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.740021    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:29:06.741972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.741951    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"]
Oct 31 03:29:06.869275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.869235    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgf6s\" (UniqueName: \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.869449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.869432    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.869500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.869483    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.971329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.971289    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.971508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.971353    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.971508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.971498    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgf6s\" (UniqueName: \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.972110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.972084    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.973851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.973825    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:06.982567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:06.982501    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgf6s\" (UniqueName: \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\") pod \"maas-api-revoke-sub-cf22eb1a-ngbqn\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:07.024327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.024284    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:07.037961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.037932    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:07.052456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.052426    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:07.337223 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:07.337192    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab966bf_6818_4368_bf68_04df7c0788f8.slice/crio-0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9 WatchSource:0}: Error finding container 0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9: Status 404 returned error can't find the container with id 0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9
Oct 31 03:29:07.339007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.338982    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"]
Oct 31 03:29:07.838068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.838019    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn" event={"ID":"0ab966bf-6818-4368-bf68-04df7c0788f8","Type":"ContainerStarted","Data":"cf8f484c6860f301cdb23c7679fdc358b5e42c4f27a54b0c27c539cb0ac24590"}
Oct 31 03:29:07.838510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.838091    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn" event={"ID":"0ab966bf-6818-4368-bf68-04df7c0788f8","Type":"ContainerStarted","Data":"0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9"}
Oct 31 03:29:07.849871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.849832    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:07.854382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:07.854329    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn" podStartSLOduration=1.854306163 podStartE2EDuration="1.854306163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:07.854008596 +0000 UTC m=+822.379228185" watchObservedRunningTime="2026-10-31 03:29:07.854306163 +0000 UTC m=+822.379525753"
Oct 31 03:29:08.847663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:08.847593    5098 generic.go:356] "Generic (PLEG): container finished" podID="0ab966bf-6818-4368-bf68-04df7c0788f8" containerID="cf8f484c6860f301cdb23c7679fdc358b5e42c4f27a54b0c27c539cb0ac24590" exitCode=0
Oct 31 03:29:08.848133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:08.847708    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn" event={"ID":"0ab966bf-6818-4368-bf68-04df7c0788f8","Type":"ContainerDied","Data":"cf8f484c6860f301cdb23c7679fdc358b5e42c4f27a54b0c27c539cb0ac24590"}
Oct 31 03:29:09.294468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:09.294429    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:29:10.027191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.027160    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:10.186728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.172790    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\" (UniqueName: \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\") pod \"0ab966bf-6818-4368-bf68-04df7c0788f8\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") "
Oct 31 03:29:10.186728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.182354    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\") pod \"0ab966bf-6818-4368-bf68-04df7c0788f8\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") "
Oct 31 03:29:10.186728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.182407    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\") pod \"0ab966bf-6818-4368-bf68-04df7c0788f8\" (UID: \"0ab966bf-6818-4368-bf68-04df7c0788f8\") "
Oct 31 03:29:10.186728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.184075    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca" pod "0ab966bf-6818-4368-bf68-04df7c0788f8" (UID: "0ab966bf-6818-4368-bf68-04df7c0788f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:10.211117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.211055    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca" pod "0ab966bf-6818-4368-bf68-04df7c0788f8" (UID: "0ab966bf-6818-4368-bf68-04df7c0788f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:10.212481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.212426    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s" pod "0ab966bf-6818-4368-bf68-04df7c0788f8" (UID: "0ab966bf-6818-4368-bf68-04df7c0788f8"). InnerVolumeSpecName "kube-api-access-hgf6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:10.290715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.290661    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:10.290715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.290701    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgf6s\" (UniqueName: \"kubernetes.io/projected/0ab966bf-6818-4368-bf68-04df7c0788f8-kube-api-access-hgf6s\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:10.290715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.290719    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ab966bf-6818-4368-bf68-04df7c0788f8-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:10.903646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.896862    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn" event={"ID":"0ab966bf-6818-4368-bf68-04df7c0788f8","Type":"ContainerDied","Data":"0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9"}
Oct 31 03:29:10.903646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.896936    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c11994590ea247da51b615dc2cb6139bd25028133fbb31ebbfb64860370c5c9"
Oct 31 03:29:10.903646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:10.897013    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"
Oct 31 03:29:12.342631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:12.341721    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:12.844371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:12.844309    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:12.963977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:12.963937    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:29:12.964288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:12.964257    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:13.691600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:13.691559    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:13.704297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:13.704275    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:13.984144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:13.984044    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:18.677456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.677424    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:29:18.678393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.678369    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0ab966bf-6818-4368-bf68-04df7c0788f8" containerName="revoke-keys"
Oct 31 03:29:18.678393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.678391    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ab966bf-6818-4368-bf68-04df7c0788f8" containerName="revoke-keys"
Oct 31 03:29:18.678642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.678525    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ab966bf-6818-4368-bf68-04df7c0788f8" containerName="revoke-keys"
Oct 31 03:29:18.683374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.683353    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.699530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.699494    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:29:18.807031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.806992    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807045    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807154    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807216    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807287    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807398    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807423    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807474    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpsnj\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.807515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.807506    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.908939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.908898    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.908958    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909023    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909066    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909184    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909215    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909244    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpsnj\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909293    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909375    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909454    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909714    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909785    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.909885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.909830    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.910246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.910224    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.911602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.911569    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.911747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.911730    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.925929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.925902    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpsnj\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:18.926288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:18.926269    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:19.006113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.006024    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:19.301522 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:19.301481    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cc0fd8_4dbc_414f_927c_ce58c7239566.slice/crio-cc7b60608d4b6da96cf01472f4861fc401629a55ccc7e7d986b8a9f126602ec2 WatchSource:0}: Error finding container cc7b60608d4b6da96cf01472f4861fc401629a55ccc7e7d986b8a9f126602ec2: Status 404 returned error can't find the container with id cc7b60608d4b6da96cf01472f4861fc401629a55ccc7e7d986b8a9f126602ec2
Oct 31 03:29:19.301895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.301871    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:29:19.304158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.304121    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:19.304288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.304206    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:19.304288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.304253    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:19.422175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.422136    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 03:29:19.427562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.427535    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.442568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.439077    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 03:29:19.517747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.517708    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.517938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.517831    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.517938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.517858    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.517938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.517911    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.518055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.517984    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdv6w\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.518055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.518009    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.518055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.518032    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.518055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.518051    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.518217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.518083    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.529092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.529053    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"]
Oct 31 03:29:19.533537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.533509    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.538257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.538229    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:19.538417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.538342    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:19.538486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.538443    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:19.538532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.538486    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:29:19.556006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.555927    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"]
Oct 31 03:29:19.619680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619637    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdv6w\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619694    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619718    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619738    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619780    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619836    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619881    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.620016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.619904    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzn9\" (UniqueName: \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.620460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620134    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620167    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620176    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620235    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.620460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620287    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620498    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620523    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620699    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.620789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.620707    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.622582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.622555    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.622673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.622624    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.631561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.631529    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.631721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.631634    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdv6w\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.721730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.721689    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.722294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.721825    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.722294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.721842    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzn9\" (UniqueName: \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.722685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.722661    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.724596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.724560    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.736883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.736853    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzn9\" (UniqueName: \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\") pod \"maas-api-revoke-sub-88c95da7-kx9cm\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.749769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.749740    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:19.788085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.787983    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:29:19.813701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.804885    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.832291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.831739    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:29:19.882671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.874556    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:19.924592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924546    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp9rs\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.924781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924677    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.924781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924736    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.924901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924787    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.924901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924855    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.924901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924878    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.925056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.924952    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.925056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.925003    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:19.925159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:19.925102    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.025049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.025001    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" event={"ID":"96cc0fd8-4dbc-414f-927c-ce58c7239566","Type":"ContainerStarted","Data":"ea3173a898ad6b11e09f287d54d47c5f92f58ed8bf9b042bbf7a8b8897dbbcb7"}
Oct 31 03:29:20.025363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.025321    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" event={"ID":"96cc0fd8-4dbc-414f-927c-ce58c7239566","Type":"ContainerStarted","Data":"cc7b60608d4b6da96cf01472f4861fc401629a55ccc7e7d986b8a9f126602ec2"}
Oct 31 03:29:20.026155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026126    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026205    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp9rs\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026262    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026313    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026359    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026422    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026446    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026516    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026568    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.026618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026588    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.027011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.026987    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.027101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.027026    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.028357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.028331    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.028559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.028535    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.036072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.030822    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.039050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.036860    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp9rs\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.039050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.039023    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.039262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.039243    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.079157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.079047    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podStartSLOduration=2.079026572 podStartE2EDuration="2.079026572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:20.07790886 +0000 UTC m=+834.603128449" watchObservedRunningTime="2026-10-31 03:29:20.079026572 +0000 UTC m=+834.604246271"
Oct 31 03:29:20.137593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.137559    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:20.184139 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:20.184111    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6c3302e_f961_43a9_b381_a9979bca3d1f.slice/crio-6aa04d70d6d7477356f84902928513e378a883d08ee52f764e91eede8a6b628e WatchSource:0}: Error finding container 6aa04d70d6d7477356f84902928513e378a883d08ee52f764e91eede8a6b628e: Status 404 returned error can't find the container with id 6aa04d70d6d7477356f84902928513e378a883d08ee52f764e91eede8a6b628e
Oct 31 03:29:20.184482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.184456    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 03:29:20.187513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.187480    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.187636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.187548    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.187636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.187579    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.232576 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:20.232539    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7dd4de3_b8e8_425a_a533_7badb0e127d6.slice/crio-c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1 WatchSource:0}: Error finding container c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1: Status 404 returned error can't find the container with id c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1
Oct 31 03:29:20.233860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.233829    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"]
Oct 31 03:29:20.488781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.487986    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.488781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.488096    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.488781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.488162    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:20.488781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:20.488563    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:29:21.011116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.010079    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:21.011843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.011808    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:21.011976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.011872    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:21.034964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.034930    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" event={"ID":"05eaf985-a4d8-4b80-9432-8397e07f3061","Type":"ContainerStarted","Data":"95ba22e14ac52439506c263760930b9fea6b4c5d823ba286aaac5630a21e1e25"}
Oct 31 03:29:21.035143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.034985    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" event={"ID":"05eaf985-a4d8-4b80-9432-8397e07f3061","Type":"ContainerStarted","Data":"f63b94dddd29e888d985dd0bf8e7f838efc11434f49e6e22d72abc365af08f7d"}
Oct 31 03:29:21.036885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.036854    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm" event={"ID":"f7dd4de3-b8e8-425a-a533-7badb0e127d6","Type":"ContainerStarted","Data":"4cfce9bfa276e3dd173517be9e2a9688a67244ac731cc1ed71fb8930d840e6bd"}
Oct 31 03:29:21.037008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.036899    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm" event={"ID":"f7dd4de3-b8e8-425a-a533-7badb0e127d6","Type":"ContainerStarted","Data":"c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1"}
Oct 31 03:29:21.038532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.038442    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" event={"ID":"d6c3302e-f961-43a9-b381-a9979bca3d1f","Type":"ContainerStarted","Data":"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"}
Oct 31 03:29:21.038532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.038488    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" event={"ID":"d6c3302e-f961-43a9-b381-a9979bca3d1f","Type":"ContainerStarted","Data":"6aa04d70d6d7477356f84902928513e378a883d08ee52f764e91eede8a6b628e"}
Oct 31 03:29:21.137984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.137927    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm" podStartSLOduration=2.137911507 podStartE2EDuration="2.137911507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.134975613 +0000 UTC m=+835.660195213" watchObservedRunningTime="2026-10-31 03:29:21.137911507 +0000 UTC m=+835.663131096"
Oct 31 03:29:21.139247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.139216    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:21.140254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.140182    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:21.140254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.140238    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:21.140484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.140434    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podStartSLOduration=2.140424571 podStartE2EDuration="2.140424571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.09835789 +0000 UTC m=+835.623577478" watchObservedRunningTime="2026-10-31 03:29:21.140424571 +0000 UTC m=+835.665644160"
Oct 31 03:29:21.164113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.164042    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podStartSLOduration=2.164020897 podStartE2EDuration="2.164020897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:21.163189875 +0000 UTC m=+835.688409465" watchObservedRunningTime="2026-10-31 03:29:21.164020897 +0000 UTC m=+835.689240500"
Oct 31 03:29:21.756692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.755686    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:21.764632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.762748    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:21.764632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:21.762822    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:22.010735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.009880    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:22.010735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.009939    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:22.049729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.049680    5098 generic.go:356] "Generic (PLEG): container finished" podID="f7dd4de3-b8e8-425a-a533-7badb0e127d6" containerID="4cfce9bfa276e3dd173517be9e2a9688a67244ac731cc1ed71fb8930d840e6bd" exitCode=0
Oct 31 03:29:22.051588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.051550    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm" event={"ID":"f7dd4de3-b8e8-425a-a533-7badb0e127d6","Type":"ContainerDied","Data":"4cfce9bfa276e3dd173517be9e2a9688a67244ac731cc1ed71fb8930d840e6bd"}
Oct 31 03:29:22.175021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.173482    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:22.294658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.265775    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:22.760720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.758075    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:22.760720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:22.758143    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:23.015689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.015168    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:23.015689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.015248    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:23.138279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.138246    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:23.138712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.138682    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:23.668958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.652927    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:23.758632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.751937    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:23.758632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.752000    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:23.851696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.851113    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\" (UniqueName: \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\") pod \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") "
Oct 31 03:29:23.851696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.851213    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\") pod \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") "
Oct 31 03:29:23.851696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.851259    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\") pod \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\" (UID: \"f7dd4de3-b8e8-425a-a533-7badb0e127d6\") "
Oct 31 03:29:23.946098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:23.946037    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca" pod "f7dd4de3-b8e8-425a-a533-7badb0e127d6" (UID: "f7dd4de3-b8e8-425a-a533-7badb0e127d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:24.042658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.040893    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9" pod "f7dd4de3-b8e8-425a-a533-7badb0e127d6" (UID: "f7dd4de3-b8e8-425a-a533-7badb0e127d6"). InnerVolumeSpecName "kube-api-access-9jzn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:24.042658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.042531    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:24.042658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.042584    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:24.045937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.044432    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca" pod "f7dd4de3-b8e8-425a-a533-7badb0e127d6" (UID: "f7dd4de3-b8e8-425a-a533-7badb0e127d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:24.045937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.045831    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jzn9\" (UniqueName: \"kubernetes.io/projected/f7dd4de3-b8e8-425a-a533-7badb0e127d6-kube-api-access-9jzn9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:24.045937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.045857    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:24.045937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.045872    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7dd4de3-b8e8-425a-a533-7badb0e127d6-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:24.141692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.141385    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:24.141692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.141448    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:24.163661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.148646    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm" event={"ID":"f7dd4de3-b8e8-425a-a533-7badb0e127d6","Type":"ContainerDied","Data":"c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1"}
Oct 31 03:29:24.163661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.148709    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c849f4d30122990d450115d246eefe7f21ef59d69fda598ddf9385cd4e0293f1"
Oct 31 03:29:24.163661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.148787    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"
Oct 31 03:29:24.753839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.753803    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:24.754131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:24.754103    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:25.011312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.011216    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:25.023632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.014698    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:25.138939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.138903    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:25.139119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.138958    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:25.770050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.770013    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:25.770645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:25.770592    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:26.007106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.007066    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 03:29:26.007353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.007324    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 03:29:26.138319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.138286    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:26.138512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.138345    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:26.751106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.751065    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body=
Oct 31 03:29:26.751382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:26.751355    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused"
Oct 31 03:29:27.138574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.138532    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 31 03:29:27.139225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.139183    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 31 03:29:27.387379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.387344    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:27.388320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.388289    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:27.389051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.388983    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:29:27.900253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.900223    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:27.901245 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.901220    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:27.902150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:27.902125    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 03:29:28.698728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:28.698697    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:28.699534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:28.699501    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:28.700307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:28.700275    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:29:31.063290 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:29:31.063252    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:29:33.151769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.151722    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"]
Oct 31 03:29:33.152394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.152375    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f7dd4de3-b8e8-425a-a533-7badb0e127d6" containerName="revoke-keys"
Oct 31 03:29:33.152394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.152394    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7dd4de3-b8e8-425a-a533-7badb0e127d6" containerName="revoke-keys"
Oct 31 03:29:33.152515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.152505    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7dd4de3-b8e8-425a-a533-7badb0e127d6" containerName="revoke-keys"
Oct 31 03:29:33.157666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.157645    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.161692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.161661    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:33.163220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.163196    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:29:33.163220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.163217    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:33.163394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.163240    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:33.165662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.165638    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"]
Oct 31 03:29:33.242869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.242831    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.243039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.242894    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55xr6\" (UniqueName: \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.243098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.243070    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.344773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.344733    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.344986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.344800    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55xr6\" (UniqueName: \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.345103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.345003    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.345973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.345943    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.347750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.347730    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.353987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.353962    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55xr6\" (UniqueName: \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\") pod \"maas-api-revoke-sub-684e3d5f-7w5v8\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.508883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.508798    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:33.704102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.704070    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:29:33.709954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.709860    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.722550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.722451    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:29:33.749691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.749655    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.749720    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.749757    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.749787    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.749833    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb4j5\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.750096    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.750178    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.750213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.750431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.750281    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.796832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.796798    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"]
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.856524    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.857187    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.857240    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.857272    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.857954    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.858035    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb4j5\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.858281    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.858369    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.858407    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.858941    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.859057    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.859204    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.859918    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.861147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.859990    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.873864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.873826    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.876550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.876516    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb4j5\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.877681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.877653    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:33.878069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:33.878050    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\") pod \"e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") " pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:34.035510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.035477    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:34.099620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.099581    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:29:34.105410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.105385    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.116886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.116857    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:29:34.163862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.163827    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkbdz\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.163893    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164092    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164118    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164170    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164209    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164245    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.164344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.164271    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.264982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.264955    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265087    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265134    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265177    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265245    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265303    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265343    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265367    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265420    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkbdz\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.265922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265532    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.266017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.265991    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.266106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.266082    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.266200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.266180    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.266285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.266242    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.267733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.267713    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.267866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.267850    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.282753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.282720    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.283328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.283305    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkbdz\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.300469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.300397    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:29:34.306816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.306790    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.318407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.318376    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:29:34.366905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.366871    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.366927    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.366973    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367081    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skggx\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367111    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367182    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367276    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367337    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.367533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.367383    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.374280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.374239    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8" event={"ID":"9e69120b-d4c3-4d80-b67d-900ea2ab5d73","Type":"ContainerStarted","Data":"38385a1547cc488e3a7e0393695676df68237da3cd0b21e950041d41924d3c01"}
Oct 31 03:29:34.374445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.374309    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8" event={"ID":"9e69120b-d4c3-4d80-b67d-900ea2ab5d73","Type":"ContainerStarted","Data":"e4876ad16532aa1476796f2a9cec9fed47a3d259401f6b0816cda1eeefea48d3"}
Oct 31 03:29:34.396691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.396593    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8" podStartSLOduration=1.396578226 podStartE2EDuration="1.396578226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:34.394287771 +0000 UTC m=+848.919507360" watchObservedRunningTime="2026-10-31 03:29:34.396578226 +0000 UTC m=+848.921797855"
Oct 31 03:29:34.410777 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:34.410737    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf91633e_99b9_43ce_a114_fecd7f1c3bb0.slice/crio-0aa95c9ff90b0f2b690e57923f158034d18499ebcbc9a4f16e041b67c40a00c2 WatchSource:0}: Error finding container 0aa95c9ff90b0f2b690e57923f158034d18499ebcbc9a4f16e041b67c40a00c2: Status 404 returned error can't find the container with id 0aa95c9ff90b0f2b690e57923f158034d18499ebcbc9a4f16e041b67c40a00c2
Oct 31 03:29:34.412363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.412336    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:29:34.414014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.413970    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.414107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.414059    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.414107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.414101    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.454659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.454538    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:34.469193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469163    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469211    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469270    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469340    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469386    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469467    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469535    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skggx\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469592    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469691    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.469936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469914    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.470000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.469975    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.470216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.470184    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.470355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.470236    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.470706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.470681    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.472173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.472146    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.472742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.472723    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.479989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.479958    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.480219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.480197    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skggx\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.639348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.639249    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:34.673355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.673320    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:29:34.679405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.679375    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.710673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.708278    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793527    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793561    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793579    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbn6p\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793618    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793660    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793847    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793867    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.793932    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.795634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.794061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.796243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.796210    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:29:34.804521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.804487    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:29:34.812406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.812356    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.812741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.812374    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:29:34.830792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.830746    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.830940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.830846    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.830940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.830889    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:34.896125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896045    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896141    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896171    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbn6p\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896214    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896251    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.896499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896282    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.896499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896345    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896375    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.896499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896418    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.896499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896474    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.896757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.896527    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.897493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.897466    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.898246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.897941    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.898715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898690    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.898827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898731    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.898827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898772    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.898827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898802    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.899003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898852    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.899003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898880    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.899003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.898901    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f5d\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.899172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.899030    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.899378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.899358    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:34.899475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.899429    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.900686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.900548    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.900779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.900699    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.907904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.907872    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbn6p\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:34.908101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:34.908082    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:35.000451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000416    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.000773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000746    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.000864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000788    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.000864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000857    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f5d\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.000970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000897    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.000970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.000921    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001093    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001132    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001202    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001266    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001534    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001598    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.001823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.001804    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.002049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.002023    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.003696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.003670    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:29:35.004110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.004089    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.005748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.005719    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.006112 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:35.006093    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod963eff1f_4fe4_4d79_ba3b_7ecd695e356c.slice/crio-0449de88a1590c66888990f37b767ab308781a1c0c6e0aa39e50dd11d7e616dd WatchSource:0}: Error finding container 0449de88a1590c66888990f37b767ab308781a1c0c6e0aa39e50dd11d7e616dd: Status 404 returned error can't find the container with id 0449de88a1590c66888990f37b767ab308781a1c0c6e0aa39e50dd11d7e616dd
Oct 31 03:29:35.009042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.009010    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.009153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.009078    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.009153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.009119    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.021116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.021079    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.021769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.021744    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f5d\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\") pod \"e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") " pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.050186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.049922    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:35.142909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.141487    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:35.407168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.405696    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 03:29:35.416208 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:35.416075    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8cf84fc_dded_4821_82d5_7ed29859c0bc.slice/crio-5c0913cc878ac0a666c7266845672064daefaf0c8e0435dd87dbb025cc1eaca5 WatchSource:0}: Error finding container 5c0913cc878ac0a666c7266845672064daefaf0c8e0435dd87dbb025cc1eaca5: Status 404 returned error can't find the container with id 5c0913cc878ac0a666c7266845672064daefaf0c8e0435dd87dbb025cc1eaca5
Oct 31 03:29:35.418227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.417743    5098 generic.go:356] "Generic (PLEG): container finished" podID="9e69120b-d4c3-4d80-b67d-900ea2ab5d73" containerID="38385a1547cc488e3a7e0393695676df68237da3cd0b21e950041d41924d3c01" exitCode=0
Oct 31 03:29:35.430297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.430259    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.431860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.431829    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.435556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.434496    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8" event={"ID":"9e69120b-d4c3-4d80-b67d-900ea2ab5d73","Type":"ContainerDied","Data":"38385a1547cc488e3a7e0393695676df68237da3cd0b21e950041d41924d3c01"}
Oct 31 03:29:35.435832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.434686    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.436949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.436922    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" event={"ID":"963eff1f-4fe4-4d79-ba3b-7ecd695e356c","Type":"ContainerStarted","Data":"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"}
Oct 31 03:29:35.441087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.441070    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 03:29:35.441296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.441280    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" event={"ID":"963eff1f-4fe4-4d79-ba3b-7ecd695e356c","Type":"ContainerStarted","Data":"0449de88a1590c66888990f37b767ab308781a1c0c6e0aa39e50dd11d7e616dd"}
Oct 31 03:29:35.441526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.441512    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" event={"ID":"df91633e-99b9-43ce-a114-fecd7f1c3bb0","Type":"ContainerStarted","Data":"3eb887d06510b6f74cea2b1d495f56e79c30d00766946cb511ec66a8bf21e4d9"}
Oct 31 03:29:35.441727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.441713    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" event={"ID":"df91633e-99b9-43ce-a114-fecd7f1c3bb0","Type":"ContainerStarted","Data":"0aa95c9ff90b0f2b690e57923f158034d18499ebcbc9a4f16e041b67c40a00c2"}
Oct 31 03:29:35.441887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.441875    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" event={"ID":"2ec62961-a725-4fd6-9ba6-70f0f7afdfee","Type":"ContainerStarted","Data":"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"}
Oct 31 03:29:35.442043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.442032    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" event={"ID":"2ec62961-a725-4fd6-9ba6-70f0f7afdfee","Type":"ContainerStarted","Data":"f1e788cdafa57a8e54e0c7b48f962201f1b63806d477a190e6176279d5e52ab1"}
Oct 31 03:29:35.442193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.434595    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.446247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.446225    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:29:35.446556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.446539    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-bcc7a4\"/\"distinct-2-w5-bcc7a4-kserve-self-signed-certs\""
Oct 31 03:29:35.467033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.466999    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:35.494729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.489047    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:35.494729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.489111    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:35.494729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.491051    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podStartSLOduration=1.491031354 podStartE2EDuration="1.491031354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.490217824 +0000 UTC m=+850.015437437" watchObservedRunningTime="2026-10-31 03:29:35.491031354 +0000 UTC m=+850.016250946"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.703845    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podStartSLOduration=2.703823877 podStartE2EDuration="2.703823877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.550667102 +0000 UTC m=+850.075886692" watchObservedRunningTime="2026-10-31 03:29:35.703823877 +0000 UTC m=+850.229043466"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.705866    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.706203    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.706338    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.706829    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.707591    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.709412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.708116    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwkcv\" (UniqueName: \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.733759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.728263    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.733759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.728368    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.733759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.728408    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:29:35.742467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.742431    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:35.802849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.802767    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:29:35.804950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.804880    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podStartSLOduration=1.8048605119999999 podStartE2EDuration="1.804860512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:35.668224256 +0000 UTC m=+850.193443866" watchObservedRunningTime="2026-10-31 03:29:35.804860512 +0000 UTC m=+850.330080102"
Oct 31 03:29:35.809681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.809470    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.809681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.809553    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwkcv\" (UniqueName: \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.812941    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.814065    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.814113    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.814147    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.814288    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.815064    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.818815    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.820138    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.822726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.822409    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.832420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.832393    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwkcv\" (UniqueName: \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\") pod \"distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") " pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:35.875897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:35.872057    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:36.045808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.041047    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:36.045894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.045848    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:36.045972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.045909    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:36.544375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.536053    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:36.544375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.536117    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:36.672633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.667225    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:36.708525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:36.708187    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:37.132802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.132765    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:37.133075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.133048    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:37.285651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.137632    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 03:29:37.322899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.322842    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:37.382048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.366954    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:37.382048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.367880    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:37.419788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.419745    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" event={"ID":"b0789148-2f28-4eab-af5b-eb1376e697ab","Type":"ContainerStarted","Data":"b08440013915e9d6100ab2dab5dc8de865394a14fe2ee7b37f4f6536a1416a4e"}
Oct 31 03:29:37.549640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.545046    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" event={"ID":"b0789148-2f28-4eab-af5b-eb1376e697ab","Type":"ContainerStarted","Data":"aaa6fbcefce3d21e802cd43fc65d4f9e231e5271caec1b5d0991080e93ec7e14"}
Oct 31 03:29:37.549640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.545107    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" event={"ID":"c8cf84fc-dded-4821-82d5-7ed29859c0bc","Type":"ContainerStarted","Data":"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"}
Oct 31 03:29:37.549640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.545152    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" event={"ID":"c8cf84fc-dded-4821-82d5-7ed29859c0bc","Type":"ContainerStarted","Data":"5c0913cc878ac0a666c7266845672064daefaf0c8e0435dd87dbb025cc1eaca5"}
Oct 31 03:29:37.628405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.621040    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:37.628405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.621119    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:37.855627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.842593    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podStartSLOduration=3.842573641 podStartE2EDuration="3.842573641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:37.458277303 +0000 UTC m=+851.983496893" watchObservedRunningTime="2026-10-31 03:29:37.842573641 +0000 UTC m=+852.367793230"
Oct 31 03:29:37.890631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.843176    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:37.890631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.881786    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:37.908543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.908469    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podStartSLOduration=3.9084491 podStartE2EDuration="3.9084491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:37.753082533 +0000 UTC m=+852.278302122" watchObservedRunningTime="2026-10-31 03:29:37.9084491 +0000 UTC m=+852.433668690"
Oct 31 03:29:38.089681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:37.980743    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:38.090359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.046754    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:38.090623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.090560    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:38.101598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.101555    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:38.101856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.101826    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:38.213986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.201851    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:38.245680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.245416    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:38.245680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.245477    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:38.273073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.273030    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:38.274853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.274818    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.339378    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.364448    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.369439    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.369476    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.695641    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.741752    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.695723    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:38.752633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.741822    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:38.760632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.757100    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:29:38.817776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.817747    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:29:38.824663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:38.818057    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:29:39.098634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.098044    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:39.148058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.144455    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:39.148058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.144517    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:39.148058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.144922    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:39.158643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.157759    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:39.158866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.158694    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:39.244646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.208143    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:39.271127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.271077    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:39.491834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.491715    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:39.492109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.492059    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:39.613632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.595515    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" event={"ID":"69fad609-0f14-4d2c-8249-6e96a8ec3d30","Type":"ContainerStarted","Data":"54df182afaff43c9caf0775e8186ef05520a61a0ae2709d1fb4f75676a73537d"}
Oct 31 03:29:39.705634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.677687    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:39.705634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.682682    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:39.705634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.690056    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:39.764483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.764363    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:29:39.779680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.765677    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:29:39.781018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.767163    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:39.843633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.767194    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:29:39.843633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:39.767215    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:29:40.025759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.019212    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\" (UniqueName: \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\") pod \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") "
Oct 31 03:29:40.081080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.081047    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\") pod \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") "
Oct 31 03:29:40.081316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.081297    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\") pod \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\" (UID: \"9e69120b-d4c3-4d80-b67d-900ea2ab5d73\") "
Oct 31 03:29:40.097113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.074763    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:40.097113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.096479    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:40.098191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.097874    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca" pod "9e69120b-d4c3-4d80-b67d-900ea2ab5d73" (UID: "9e69120b-d4c3-4d80-b67d-900ea2ab5d73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:40.103652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.102979    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca" pod "9e69120b-d4c3-4d80-b67d-900ea2ab5d73" (UID: "9e69120b-d4c3-4d80-b67d-900ea2ab5d73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:40.106461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.105409    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:40.121624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.112875    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6" pod "9e69120b-d4c3-4d80-b67d-900ea2ab5d73" (UID: "9e69120b-d4c3-4d80-b67d-900ea2ab5d73"). InnerVolumeSpecName "kube-api-access-55xr6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:40.121624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.112984    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:40.121624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.113030    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:40.200724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.165677    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:40.325632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.315693    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:40.325632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.195651    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:40.325632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.316223    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55xr6\" (UniqueName: \"kubernetes.io/projected/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-kube-api-access-55xr6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:40.325632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.316248    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e69120b-d4c3-4d80-b67d-900ea2ab5d73-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:40.365096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.365058    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:29:40.365291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.365150    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:40.443207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.443174    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:29:40.499181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.499144    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:40.499455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.499420    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:40.667862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.667752    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:40.830899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.830818    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:40.831464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.787760    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:40.852996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.787808    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:29:40.853964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.803496    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:40.854345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.765752    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:29:40.926215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.926110    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8" event={"ID":"9e69120b-d4c3-4d80-b67d-900ea2ab5d73","Type":"ContainerDied","Data":"e4876ad16532aa1476796f2a9cec9fed47a3d259401f6b0816cda1eeefea48d3"}
Oct 31 03:29:40.926483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.926461    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4876ad16532aa1476796f2a9cec9fed47a3d259401f6b0816cda1eeefea48d3"
Oct 31 03:29:40.926692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.926675    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"
Oct 31 03:29:40.952713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:40.952670    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 31 03:29:41.071630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.035818    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:41.185726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.184341    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:41.185726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.099395    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:41.185726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.185482    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:41.185726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.106058    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:41.186075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.146668    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:41.186075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.185863    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:41.416747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.416697    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:41.627022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.626984    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:41.751338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.751293    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:41.767888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.725513    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:41.768164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:41.768131    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:42.185695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.027055    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:42.185695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.028177    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:29:42.268632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.236397    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:29:42.316943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.316885    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:29:42.565635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.336419    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:42.565635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.563453    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:42.599979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:42.599926    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:29:43.013693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.001421    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.236534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.236488    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:43.237623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.078028    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 31 03:29:43.237843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.093794    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:43.238070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.039950    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:43.256865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.109935    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.257193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.257153    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:43.257448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.170169    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.257640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.257618    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:43.257829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.174541    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.257986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.257965    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:43.258125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.174795    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.258236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.258218    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:43.342641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.339086    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:29:43.389632 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:29:43.387382    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.641s"
Oct 31 03:29:43.389632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.387631    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:29:43.389632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.387673    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:43.410635 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:29:43.409191    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e69120b_d4c3_4d80_b67d_900ea2ab5d73.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:29:43.538133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.538091    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.538391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.538361    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:43.564668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.555842    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.564668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.555910    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:43.565601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.565558    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:43.600920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.600817    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:29:43.694781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.694727    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:43.695090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:43.695035    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:44.004581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.004461    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:44.116633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.109843    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:44.116633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.109914    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:44.120827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.120790    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:44.121082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.121053    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:44.163773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.162838    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:44.163773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.162915    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:44.255853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.255731    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:29:44.487649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.360291    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:44.487649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.382662    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:29:44.526692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.518401    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:44.684732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.684685    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:44.684972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.605992    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:44.685416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.566726    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:44.686657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.658435    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:44.686827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.686801    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:44.707733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.671778    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:29:44.724631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:44.708943    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:45.024844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.024730    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:45.053726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.042500    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:45.053726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.042573    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:45.053726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.052548    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:45.053726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.052702    5098 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 03:29:45.133884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.133842    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:45.134745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.134698    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:45.157818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.157775    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:45.183767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.183715    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:45.414084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.414022    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 31 03:29:45.432064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.417992    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:29:45.433215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.430466    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 03:29:45.481630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.481566    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body=
Oct 31 03:29:45.481826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.481645    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused"
Oct 31 03:29:45.505733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.486906    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:29:45.505733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.487771    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:29:45.527684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.527652    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:29:45.665727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.655836    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Oct 31 03:29:45.665727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.655899    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Oct 31 03:29:45.699502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.698928    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:45.887180 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.878890    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:29:45.893058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:45.893009    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:46.104633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.039547    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 03:29:46.130770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.130717    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 03:29:46.131051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.099556    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:46.131224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.131199    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:46.164857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.164816    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:46.190758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.190699    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:46.708691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.602842    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:46.710728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:46.710679    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:47.078632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.070863    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:47.101637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.097889    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Oct 31 03:29:47.101637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.098045    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Oct 31 03:29:47.205634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.201824    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:29:47.205634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.204402    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:47.232633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.211043    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:47.232633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.211103    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:47.232633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.211434    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:47.288633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.274548    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:29:47.336721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.336628    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:29:47.618366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.402863    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podStartSLOduration=12.402839742 podStartE2EDuration="12.402839742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:47.337527287 +0000 UTC m=+861.862746891" watchObservedRunningTime="2026-10-31 03:29:47.402839742 +0000 UTC m=+861.928059326"
Oct 31 03:29:47.712219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.711739    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:47.754900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.753729    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:29:47.754900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.753811    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:47.820444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.820371    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:29:47.877517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.877423    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:47.908631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.908548    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:47.932636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.912551    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:47.932636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.916889    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:29:47.934299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:47.934265    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:29:48.143046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.142961    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 03:29:48.143677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.143050    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 03:29:48.276319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.266206    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" event={"ID":"69fad609-0f14-4d2c-8249-6e96a8ec3d30","Type":"ContainerStarted","Data":"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"}
Oct 31 03:29:48.331533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.331491    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"]
Oct 31 03:29:48.332197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.332176    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9e69120b-d4c3-4d80-b67d-900ea2ab5d73" containerName="revoke-keys"
Oct 31 03:29:48.332197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.332195    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e69120b-d4c3-4d80-b67d-900ea2ab5d73" containerName="revoke-keys"
Oct 31 03:29:48.332383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.332337    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e69120b-d4c3-4d80-b67d-900ea2ab5d73" containerName="revoke-keys"
Oct 31 03:29:48.337167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.336314    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.341926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.341896    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:29:48.343040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.342146    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:29:48.343282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.342885    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:29:48.343703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.342967    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:29:48.345094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.344900    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"]
Oct 31 03:29:48.505289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.505182    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qspf\" (UniqueName: \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.505289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.505240    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.505538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.505419    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.518796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.518758    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:48.519883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.519836    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:48.521744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.521721    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:29:48.606500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.606465    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.607317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.606754    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qspf\" (UniqueName: \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.607317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.606817    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.607708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.607682    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.609954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.609931    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.626769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.626734    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qspf\" (UniqueName: \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\") pod \"maas-api-revoke-sub-6a2b2019-phghv\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.663637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.663593    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:48.991646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:48.991438    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"]
Oct 31 03:29:49.097453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.096751    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:29:49.097453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.096817    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:49.278064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.277926    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv" event={"ID":"29795917-d8b8-4d77-be74-c20e6ee03f5a","Type":"ContainerStarted","Data":"b73f17c7655010e54986dd4bb18fcf2b10ab85ac09f266479d2432506f39ca70"}
Oct 31 03:29:49.660854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.660768    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:49.661160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.661134    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:49.662103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.662073    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:29:49.754627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.746930    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:49.757235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.757190    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:49.760740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:49.760710    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:29:50.289292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:50.289261    5098 generic.go:356] "Generic (PLEG): container finished" podID="29795917-d8b8-4d77-be74-c20e6ee03f5a" containerID="00ce380737c9387f45c7ca4a615c524a5f8a78f8d3c47a190ce4de8af5f8fc1e" exitCode=0
Oct 31 03:29:50.289740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:50.289342    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv" event={"ID":"29795917-d8b8-4d77-be74-c20e6ee03f5a","Type":"ContainerDied","Data":"00ce380737c9387f45c7ca4a615c524a5f8a78f8d3c47a190ce4de8af5f8fc1e"}
Oct 31 03:29:51.535448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.535418    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:51.582173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.582138    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\") pod \"29795917-d8b8-4d77-be74-c20e6ee03f5a\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") "
Oct 31 03:29:51.582361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.582189    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\" (UniqueName: \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\") pod \"29795917-d8b8-4d77-be74-c20e6ee03f5a\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") "
Oct 31 03:29:51.582423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.582406    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\") pod \"29795917-d8b8-4d77-be74-c20e6ee03f5a\" (UID: \"29795917-d8b8-4d77-be74-c20e6ee03f5a\") "
Oct 31 03:29:51.582819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.582785    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca" pod "29795917-d8b8-4d77-be74-c20e6ee03f5a" (UID: "29795917-d8b8-4d77-be74-c20e6ee03f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:51.583258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.583232    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:51.584483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.584456    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca" pod "29795917-d8b8-4d77-be74-c20e6ee03f5a" (UID: "29795917-d8b8-4d77-be74-c20e6ee03f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:51.584864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.584846    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf" pod "29795917-d8b8-4d77-be74-c20e6ee03f5a" (UID: "29795917-d8b8-4d77-be74-c20e6ee03f5a"). InnerVolumeSpecName "kube-api-access-2qspf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:51.684588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.684535    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29795917-d8b8-4d77-be74-c20e6ee03f5a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:51.684588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:51.684582    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qspf\" (UniqueName: \"kubernetes.io/projected/29795917-d8b8-4d77-be74-c20e6ee03f5a-kube-api-access-2qspf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:52.106325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.106279    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"]
Oct 31 03:29:52.107109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.107085    5098 cpu_manager.go:414] "containerMap: removing container" podUID="29795917-d8b8-4d77-be74-c20e6ee03f5a" containerName="revoke-keys"
Oct 31 03:29:52.107109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.107108    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29795917-d8b8-4d77-be74-c20e6ee03f5a" containerName="revoke-keys"
Oct 31 03:29:52.107307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.107268    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="29795917-d8b8-4d77-be74-c20e6ee03f5a" containerName="revoke-keys"
Oct 31 03:29:52.111336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.111308    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.115732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.115700    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"]
Oct 31 03:29:52.197019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.196981    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qlgb\" (UniqueName: \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.197242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.197217    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.197519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.197499    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.299372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.299329    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.299556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.299382    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qlgb\" (UniqueName: \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.299556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.299504    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.300344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.300322    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.301904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.301883    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.307892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.307863    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qlgb\" (UniqueName: \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\") pod \"maas-api-revoke-sub-0099b050-gjzfz\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.342731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.342700    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"
Oct 31 03:29:52.342911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.342704    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv" event={"ID":"29795917-d8b8-4d77-be74-c20e6ee03f5a","Type":"ContainerDied","Data":"b73f17c7655010e54986dd4bb18fcf2b10ab85ac09f266479d2432506f39ca70"}
Oct 31 03:29:52.342911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.342832    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b73f17c7655010e54986dd4bb18fcf2b10ab85ac09f266479d2432506f39ca70"
Oct 31 03:29:52.430583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.430506    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:52.784466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:52.783701    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"]
Oct 31 03:29:53.353961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:53.353859    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz" event={"ID":"eeac5ee5-80ff-4430-aaee-3441dac5062a","Type":"ContainerStarted","Data":"1a6b631278c0a00705a9cb6c727b3eb2cc93d13fe93d071cf23b941302a8f25e"}
Oct 31 03:29:53.354164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:53.353960    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz" event={"ID":"eeac5ee5-80ff-4430-aaee-3441dac5062a","Type":"ContainerStarted","Data":"14881f7350b7ea841de708cef54a8f62fdac31a8cdb6a5517a3661afeaea8fcd"}
Oct 31 03:29:53.371579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:53.371525    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz" podStartSLOduration=1.371509611 podStartE2EDuration="1.371509611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:29:53.370363504 +0000 UTC m=+867.895583107" watchObservedRunningTime="2026-10-31 03:29:53.371509611 +0000 UTC m=+867.896729200"
Oct 31 03:29:54.363714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:54.363679    5098 generic.go:356] "Generic (PLEG): container finished" podID="eeac5ee5-80ff-4430-aaee-3441dac5062a" containerID="1a6b631278c0a00705a9cb6c727b3eb2cc93d13fe93d071cf23b941302a8f25e" exitCode=0
Oct 31 03:29:54.364168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:54.363779    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz" event={"ID":"eeac5ee5-80ff-4430-aaee-3441dac5062a","Type":"ContainerDied","Data":"1a6b631278c0a00705a9cb6c727b3eb2cc93d13fe93d071cf23b941302a8f25e"}
Oct 31 03:29:55.336047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.336000    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"]
Oct 31 03:29:55.344673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.344644    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.351677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.351648    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"]
Oct 31 03:29:55.583148 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.583111    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg7xt\" (UniqueName: \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.583654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.583157    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.583654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.583372    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.643348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.643315    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:55.685108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.685072    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.685268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.685185    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.686067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.685446    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg7xt\" (UniqueName: \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.686214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.686070    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.688089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.688055    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.694547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.694519    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg7xt\" (UniqueName: \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\") pod \"maas-api-revoke-sub-826ba39f-6qnxc\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:55.786267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.786231    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\") pod \"eeac5ee5-80ff-4430-aaee-3441dac5062a\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") "
Oct 31 03:29:55.786469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.786294    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\") pod \"eeac5ee5-80ff-4430-aaee-3441dac5062a\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") "
Oct 31 03:29:55.786469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.786363    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\" (UniqueName: \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\") pod \"eeac5ee5-80ff-4430-aaee-3441dac5062a\" (UID: \"eeac5ee5-80ff-4430-aaee-3441dac5062a\") "
Oct 31 03:29:55.787185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.787123    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca" pod "eeac5ee5-80ff-4430-aaee-3441dac5062a" (UID: "eeac5ee5-80ff-4430-aaee-3441dac5062a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:29:55.788883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.788828    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca" pod "eeac5ee5-80ff-4430-aaee-3441dac5062a" (UID: "eeac5ee5-80ff-4430-aaee-3441dac5062a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:29:55.790330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.790299    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb" pod "eeac5ee5-80ff-4430-aaee-3441dac5062a" (UID: "eeac5ee5-80ff-4430-aaee-3441dac5062a"). InnerVolumeSpecName "kube-api-access-5qlgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:29:55.872746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.872661    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:55.885334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.885294    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:55.888494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.888474    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:55.888494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.888497    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eeac5ee5-80ff-4430-aaee-3441dac5062a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:55.888627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.888508    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qlgb\" (UniqueName: \"kubernetes.io/projected/eeac5ee5-80ff-4430-aaee-3441dac5062a-kube-api-access-5qlgb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:29:55.970633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:55.969972    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:29:56.339456 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:29:56.339421    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1eb1148_4308_44bf_a06f_57bc4d345c42.slice/crio-0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c WatchSource:0}: Error finding container 0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c: Status 404 returned error can't find the container with id 0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c
Oct 31 03:29:56.340843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.340820    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"]
Oct 31 03:29:56.384328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.384297    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc" event={"ID":"f1eb1148-4308-44bf-a06f-57bc4d345c42","Type":"ContainerStarted","Data":"0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c"}
Oct 31 03:29:56.386009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.385990    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"
Oct 31 03:29:56.386144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.386023    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz" event={"ID":"eeac5ee5-80ff-4430-aaee-3441dac5062a","Type":"ContainerDied","Data":"14881f7350b7ea841de708cef54a8f62fdac31a8cdb6a5517a3661afeaea8fcd"}
Oct 31 03:29:56.386144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.386067    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14881f7350b7ea841de708cef54a8f62fdac31a8cdb6a5517a3661afeaea8fcd"
Oct 31 03:29:56.397691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.397669    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:29:56.861969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:56.861932    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:29:57.397746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:57.397710    5098 generic.go:356] "Generic (PLEG): container finished" podID="f1eb1148-4308-44bf-a06f-57bc4d345c42" containerID="d0ab6b414535b58cb6ef7be4e298afe23ed83ba13a7808ae5354245848ba1e9d" exitCode=0
Oct 31 03:29:57.397943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:57.397800    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc" event={"ID":"f1eb1148-4308-44bf-a06f-57bc4d345c42","Type":"ContainerDied","Data":"d0ab6b414535b58cb6ef7be4e298afe23ed83ba13a7808ae5354245848ba1e9d"}
Oct 31 03:29:59.548765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:59.548714    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:29:59.867550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:59.867449    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:00.103950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:59.867814    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:30:00.117440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:59.868317    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:30:00.136128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:29:59.991714    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:00.136655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.048713    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:00.449725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.447146    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:30:00.567819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.567760    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:00.608902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.608864    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:30:00.710689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.612856    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\" (UniqueName: \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\") pod \"f1eb1148-4308-44bf-a06f-57bc4d345c42\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") "
Oct 31 03:30:00.710689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.612979    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\") pod \"f1eb1148-4308-44bf-a06f-57bc4d345c42\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") "
Oct 31 03:30:00.710689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.613026    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\") pod \"f1eb1148-4308-44bf-a06f-57bc4d345c42\" (UID: \"f1eb1148-4308-44bf-a06f-57bc4d345c42\") "
Oct 31 03:30:00.745944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.745884    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt" pod "f1eb1148-4308-44bf-a06f-57bc4d345c42" (UID: "f1eb1148-4308-44bf-a06f-57bc4d345c42"). InnerVolumeSpecName "kube-api-access-dg7xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:00.766630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.749751    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg7xt\" (UniqueName: \"kubernetes.io/projected/f1eb1148-4308-44bf-a06f-57bc4d345c42-kube-api-access-dg7xt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:00.775881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.775838    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca" pod "f1eb1148-4308-44bf-a06f-57bc4d345c42" (UID: "f1eb1148-4308-44bf-a06f-57bc4d345c42"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:00.815633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.809843    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca" pod "f1eb1148-4308-44bf-a06f-57bc4d345c42" (UID: "f1eb1148-4308-44bf-a06f-57bc4d345c42"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:00.979701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:00.976064    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:01.149737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.065803    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:01.152046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.152013    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1eb1148-4308-44bf-a06f-57bc4d345c42-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:01.259427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.129392    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:01.151941    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.534s"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.284225    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.284283    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.176568    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.129515    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.260176    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.152561    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.176360    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.365583    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f1eb1148-4308-44bf-a06f-57bc4d345c42" containerName="revoke-keys"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.365635    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1eb1148-4308-44bf-a06f-57bc4d345c42" containerName="revoke-keys"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.365674    5098 cpu_manager.go:414] "containerMap: removing container" podUID="eeac5ee5-80ff-4430-aaee-3441dac5062a" containerName="revoke-keys"
Oct 31 03:30:01.369620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.365683    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeac5ee5-80ff-4430-aaee-3441dac5062a" containerName="revoke-keys"
Oct 31 03:30:01.501847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.501805    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1eb1148-4308-44bf-a06f-57bc4d345c42" containerName="revoke-keys"
Oct 31 03:30:01.502096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.502077    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeac5ee5-80ff-4430-aaee-3441dac5062a" containerName="revoke-keys"
Oct 31 03:30:01.571158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.571098    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:01.716471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.716382    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:01.861990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.861875    5098 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-6jnt8 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:30:01.862317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:01.862281    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8" podUID="5d7ff043-ea26-4c96-874d-c2d6ab18b6a9" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.327843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.327776    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.354576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.354533    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.461634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.411216    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.520127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.411279    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:30:02.530829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.411312    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.675166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.446911    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:02.709785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.447020    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:02.748968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.748914    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:02.749403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.674652    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnjk5\" (UniqueName: \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:02.775040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.774947    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:02.801132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.572819    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:30:02.802168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.802126    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Oct 31 03:30:02.803991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.803950    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:30:02.804293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.804257    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:30:02.973812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:02.973708    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.023655 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:02.992666    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:30:03.023655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.016082    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:30:03.023655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.016257    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:30:03.123070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.095973    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:03.123070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.096230    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnjk5\" (UniqueName: \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:03.123070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.097837    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:03.158949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.158910    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:30:03.191640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.181898    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.352762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.352705    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.445634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.430903    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:30:03.469631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.448076    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwbdb\" (UniqueName: \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.562627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.555139    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.562627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.523015    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.584646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.523057    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:03.615640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.476022    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.733667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.698166    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.733667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.700063    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.733667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.700213    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwbdb\" (UniqueName: \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.798730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.750684    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:03.798730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.751540    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnjk5\" (UniqueName: \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\") pod \"maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:03.805102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.805048    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.826562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.823059    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:03.909330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:03.909284    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:04.055772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.052720    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:30:04.055772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.052790    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.375686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.375563    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:30:04.479491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.450747    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwbdb\" (UniqueName: \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:04.603532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.603487    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:04.608651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.604511    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.650533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.650424    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.706237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.692853    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:30:04.706237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.692924    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.737544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.728585    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.784163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.784107    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:04.817717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.814194    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:30:04.847779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.847733    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:30:04.889130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:04.889091    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:05.131347 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:05.131309    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.306s"
Oct 31 03:30:05.131728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.131702    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc" event={"ID":"f1eb1148-4308-44bf-a06f-57bc4d345c42","Type":"ContainerDied","Data":"0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c"}
Oct 31 03:30:05.133206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.133178    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e3b3566ef04e1224ac8634ed3a17b27b4a715fcfdcb8f42f37207888e55823c"
Oct 31 03:30:05.133472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.133453    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:05.223931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.223091    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:05.223931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.223158    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:05.223931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.223233    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:30:05.223931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.223253    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:05.229251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.229220    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:05.235633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.235586    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.235916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.235884    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.268015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.267850    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.267038    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268288    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268342    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268365    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268404    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268437    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268489    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268508    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268542    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:30:05.269211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.268575    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:30:05.269892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.269532    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:30:05.269892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.269827    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:30:05.269892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.269880    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:30:05.270042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.269958    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:30:05.342389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.340078    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:30:05.348129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.348102    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.350347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.350320    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:30:05.359097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.358961    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.359215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.359129    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.359215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.359208    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd66\" (UniqueName: \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.360715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.359447    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.360715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.359692    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8nc6\" (UniqueName: \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.361144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.361108    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465232    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd66\" (UniqueName: \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465323    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465366    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465422    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465492    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465550    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465601    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465709    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvjk2\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465778    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465842    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8nc6\" (UniqueName: \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.465952    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.466048    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.466096    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.466223    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.471932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.467165    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.472873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.472306    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.481209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.481072    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.499663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.494392    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8nc6\" (UniqueName: \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\") pod \"maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.499663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.499044    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") pod \"maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.504877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.504838    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd66\" (UniqueName: \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.569011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.568654    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.569011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.568752    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.569011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.568907    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.569011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.568939    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.569011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.568992    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.569096    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.570323    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.570439    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvjk2\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.570503    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.571111    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.571394    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.573005    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.573466    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.573837    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.574843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.574115    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.582408 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:05.582374    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933 WatchSource:0}: Error finding container 8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933: Status 404 returned error can't find the container with id 8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933
Oct 31 03:30:05.584133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.584106    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.587938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.587905    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.589159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.589132    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.595255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.595117    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvjk2\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\") pod \"e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") " pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.601765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.601739    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:05.603079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.603051    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:05.656107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.656070    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:05.675562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.671853    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:05.682596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.682556    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:05.841862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.841817    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerStarted","Data":"8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933"}
Oct 31 03:30:05.844727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.844593    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerStarted","Data":"d366c404ce3d6b92167562368f3115e1591ea7dd6f4a4cd32a34c5b1f4737a8e"}
Oct 31 03:30:05.924764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:05.924733    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:05.927216 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:05.927184    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57836d63_4df6_41ff_85a4_3e2aa6768a6f.slice/crio-37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863 WatchSource:0}: Error finding container 37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863: Status 404 returned error can't find the container with id 37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863
Oct 31 03:30:06.104519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.104481    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:30:06.110410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.110378    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:06.116116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.116086    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:06.157341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.157178    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:06.157341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.157262    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:06.157341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.157294    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:06.856141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.856101    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerStarted","Data":"97e126d49b9b4913c4384a93c6533b9ef827e775cbfb73f049ed7b5ccac690ee"}
Oct 31 03:30:06.858171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.858141    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerStarted","Data":"8250a11b8036ce76ed0263192d21cd7a8e9545dc505f98ae1a2f6c94c2b6c1d5"}
Oct 31 03:30:06.858309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.858197    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerStarted","Data":"37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863"}
Oct 31 03:30:06.860275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.860241    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" event={"ID":"ee2dd46e-8261-4689-a7db-a708bf76fc0e","Type":"ContainerStarted","Data":"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"}
Oct 31 03:30:06.860392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.860309    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" event={"ID":"ee2dd46e-8261-4689-a7db-a708bf76fc0e","Type":"ContainerStarted","Data":"e01f4914b261a9442a518a880824d1602acc5aa707d146503017351e24e2a13b"}
Oct 31 03:30:06.862808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.862785    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerStarted","Data":"caf70ffdb17aea10f33c50a4908f7f16d061aa7fc595a45f111d9436dcb5916f"}
Oct 31 03:30:06.862915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.862830    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerStarted","Data":"c4d60a286ff23e97ff6b872c47d7d031f388e2d0b755ed3640a17be6679cb754"}
Oct 31 03:30:06.864684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.864656    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerStarted","Data":"db901ff68e26c53df797706aa73d58baf98fea04195dc40b4e20d89cb08aa5a6"}
Oct 31 03:30:06.864793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.864713    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerStarted","Data":"40c4dd694306a72c2a2d3adcfa4788d032049739cf1c5008f8783d8806cc9011"}
Oct 31 03:30:06.866564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.866535    5098 generic.go:356] "Generic (PLEG): container finished" podID="98a7637d-8306-4f42-beb4-6452b1174e78" containerID="d6afa41bd0ea635710daa15bc8e355bade009a36b8a06e67a7a87d5fde37ee57" exitCode=60
Oct 31 03:30:06.866679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.866640    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerDied","Data":"d6afa41bd0ea635710daa15bc8e355bade009a36b8a06e67a7a87d5fde37ee57"}
Oct 31 03:30:06.866853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.866837    5098 scope.go:122] "RemoveContainer" containerID="d6afa41bd0ea635710daa15bc8e355bade009a36b8a06e67a7a87d5fde37ee57"
Oct 31 03:30:06.880985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.880932    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" podStartSLOduration=6.880915725 podStartE2EDuration="6.880915725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.876680337 +0000 UTC m=+881.401899927" watchObservedRunningTime="2026-10-31 03:30:06.880915725 +0000 UTC m=+881.406135313"
Oct 31 03:30:06.898399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.898338    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" podStartSLOduration=6.898322337 podStartE2EDuration="6.898322337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.893055384 +0000 UTC m=+881.418274975" watchObservedRunningTime="2026-10-31 03:30:06.898322337 +0000 UTC m=+881.423541967"
Oct 31 03:30:06.917188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.917131    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podStartSLOduration=2.917116939 podStartE2EDuration="2.917116939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.912242566 +0000 UTC m=+881.437462165" watchObservedRunningTime="2026-10-31 03:30:06.917116939 +0000 UTC m=+881.442336613"
Oct 31 03:30:06.930201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.930137    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" podStartSLOduration=6.930121262 podStartE2EDuration="6.930121262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.927924455 +0000 UTC m=+881.453144040" watchObservedRunningTime="2026-10-31 03:30:06.930121262 +0000 UTC m=+881.455340850"
Oct 31 03:30:06.967814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:06.967753    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" podStartSLOduration=6.967733509 podStartE2EDuration="6.967733509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:06.965810432 +0000 UTC m=+881.491030022" watchObservedRunningTime="2026-10-31 03:30:06.967733509 +0000 UTC m=+881.492953097"
Oct 31 03:30:07.684274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.684240    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:07.685795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.685761    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:07.685943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.685825    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:07.891211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.891174    5098 generic.go:356] "Generic (PLEG): container finished" podID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerID="97e126d49b9b4913c4384a93c6533b9ef827e775cbfb73f049ed7b5ccac690ee" exitCode=60
Oct 31 03:30:07.891678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.891268    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerDied","Data":"97e126d49b9b4913c4384a93c6533b9ef827e775cbfb73f049ed7b5ccac690ee"}
Oct 31 03:30:07.891678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.891625    5098 scope.go:122] "RemoveContainer" containerID="97e126d49b9b4913c4384a93c6533b9ef827e775cbfb73f049ed7b5ccac690ee"
Oct 31 03:30:07.893287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.893263    5098 generic.go:356] "Generic (PLEG): container finished" podID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerID="8250a11b8036ce76ed0263192d21cd7a8e9545dc505f98ae1a2f6c94c2b6c1d5" exitCode=60
Oct 31 03:30:07.893407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.893380    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerDied","Data":"8250a11b8036ce76ed0263192d21cd7a8e9545dc505f98ae1a2f6c94c2b6c1d5"}
Oct 31 03:30:07.893709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.893689    5098 scope.go:122] "RemoveContainer" containerID="8250a11b8036ce76ed0263192d21cd7a8e9545dc505f98ae1a2f6c94c2b6c1d5"
Oct 31 03:30:07.895621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.895558    5098 generic.go:356] "Generic (PLEG): container finished" podID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerID="caf70ffdb17aea10f33c50a4908f7f16d061aa7fc595a45f111d9436dcb5916f" exitCode=60
Oct 31 03:30:07.895707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.895655    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerDied","Data":"caf70ffdb17aea10f33c50a4908f7f16d061aa7fc595a45f111d9436dcb5916f"}
Oct 31 03:30:07.895945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.895925    5098 scope.go:122] "RemoveContainer" containerID="caf70ffdb17aea10f33c50a4908f7f16d061aa7fc595a45f111d9436dcb5916f"
Oct 31 03:30:07.897749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.897725    5098 generic.go:356] "Generic (PLEG): container finished" podID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerID="db901ff68e26c53df797706aa73d58baf98fea04195dc40b4e20d89cb08aa5a6" exitCode=60
Oct 31 03:30:07.897854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.897808    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerDied","Data":"db901ff68e26c53df797706aa73d58baf98fea04195dc40b4e20d89cb08aa5a6"}
Oct 31 03:30:07.898050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.898035    5098 scope.go:122] "RemoveContainer" containerID="db901ff68e26c53df797706aa73d58baf98fea04195dc40b4e20d89cb08aa5a6"
Oct 31 03:30:07.900819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.900797    5098 scope.go:122] "RemoveContainer" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"
Oct 31 03:30:07.900972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:07.900795    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerStarted","Data":"38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"}
Oct 31 03:30:07.901131 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:07.901107    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685_odh-ai-gateway-infra(98a7637d-8306-4f42-beb4-6452b1174e78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" podUID="98a7637d-8306-4f42-beb4-6452b1174e78"
Oct 31 03:30:08.684397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.684357    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:08.684685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.684426    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:08.912671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.912637    5098 generic.go:356] "Generic (PLEG): container finished" podID="98a7637d-8306-4f42-beb4-6452b1174e78" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2" exitCode=60
Oct 31 03:30:08.913195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.912744    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerDied","Data":"38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"}
Oct 31 03:30:08.913195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.912806    5098 scope.go:122] "RemoveContainer" containerID="d6afa41bd0ea635710daa15bc8e355bade009a36b8a06e67a7a87d5fde37ee57"
Oct 31 03:30:08.913195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.913151    5098 scope.go:122] "RemoveContainer" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"
Oct 31 03:30:08.913558 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:08.913533    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685_odh-ai-gateway-infra(98a7637d-8306-4f42-beb4-6452b1174e78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" podUID="98a7637d-8306-4f42-beb4-6452b1174e78"
Oct 31 03:30:08.917272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.916860    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerStarted","Data":"2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"}
Oct 31 03:30:08.921617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:08.919825    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerStarted","Data":"4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"}
Oct 31 03:30:09.006628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.004693    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerStarted","Data":"98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"}
Oct 31 03:30:09.023019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.022936    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerStarted","Data":"f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"}
Oct 31 03:30:09.683910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.683871    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:09.684111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.683936    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:09.811646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.808660    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"]
Oct 31 03:30:09.830635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.822036    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:09.830635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.828365    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"]
Oct 31 03:30:09.838638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.837212    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:30:09.902252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.902217    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkd6q\" (UniqueName: \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:09.902545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.902524    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:09.902828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:09.902810    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.004470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.004381    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.004889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.004518    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkd6q\" (UniqueName: \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.004889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.004592    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.005270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.005245    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.006862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.006831    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.013335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.013305    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkd6q\" (UniqueName: \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\") pod \"maas-api-revoke-sub-c438cdea-z2wnf\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.036336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.033704    5098 generic.go:356] "Generic (PLEG): container finished" podID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerID="f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600" exitCode=60
Oct 31 03:30:10.036336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.033911    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerDied","Data":"f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"}
Oct 31 03:30:10.036336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.033979    5098 scope.go:122] "RemoveContainer" containerID="db901ff68e26c53df797706aa73d58baf98fea04195dc40b4e20d89cb08aa5a6"
Oct 31 03:30:10.036336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.034587    5098 scope.go:122] "RemoveContainer" containerID="f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"
Oct 31 03:30:10.036336 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:10.035071    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6_odh-ai-gateway-infra(b7684d2d-86c9-4cbc-8c1f-fb6645720deb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb"
Oct 31 03:30:10.037164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.036885    5098 scope.go:122] "RemoveContainer" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"
Oct 31 03:30:10.037236 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:10.037214    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685_odh-ai-gateway-infra(98a7637d-8306-4f42-beb4-6452b1174e78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" podUID="98a7637d-8306-4f42-beb4-6452b1174e78"
Oct 31 03:30:10.040309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.040279    5098 generic.go:356] "Generic (PLEG): container finished" podID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerID="2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e" exitCode=60
Oct 31 03:30:10.040841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.040418    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerDied","Data":"2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"}
Oct 31 03:30:10.040929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.040870    5098 scope.go:122] "RemoveContainer" containerID="2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"
Oct 31 03:30:10.041303 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:10.041275    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf_odh-ai-gateway-infra(1d89dbe9-76aa-4dd6-91d2-7e02adeb841a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a"
Oct 31 03:30:10.043398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.043372    5098 generic.go:356] "Generic (PLEG): container finished" podID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerID="4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e" exitCode=60
Oct 31 03:30:10.043558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.043482    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerDied","Data":"4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"}
Oct 31 03:30:10.043908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.043890    5098 scope.go:122] "RemoveContainer" containerID="4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"
Oct 31 03:30:10.044244 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:10.044225    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn_odh-ai-gateway-infra(57836d63-4df6-41ff-85a4-3e2aa6768a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f"
Oct 31 03:30:10.045828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.045803    5098 generic.go:356] "Generic (PLEG): container finished" podID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerID="98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439" exitCode=60
Oct 31 03:30:10.045924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.045836    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerDied","Data":"98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"}
Oct 31 03:30:10.046214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.046194    5098 scope.go:122] "RemoveContainer" containerID="98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"
Oct 31 03:30:10.046671 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:10.046648    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v_odh-ai-gateway-infra(47b7d5d3-30a5-49f6-a1d0-8be2b89795bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb"
Oct 31 03:30:10.082769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.082737    5098 scope.go:122] "RemoveContainer" containerID="97e126d49b9b4913c4384a93c6533b9ef827e775cbfb73f049ed7b5ccac690ee"
Oct 31 03:30:10.099667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.099642    5098 scope.go:122] "RemoveContainer" containerID="8250a11b8036ce76ed0263192d21cd7a8e9545dc505f98ae1a2f6c94c2b6c1d5"
Oct 31 03:30:10.115775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.115754    5098 scope.go:122] "RemoveContainer" containerID="caf70ffdb17aea10f33c50a4908f7f16d061aa7fc595a45f111d9436dcb5916f"
Oct 31 03:30:10.206553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.206517    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:10.520219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.520186    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"]
Oct 31 03:30:10.684254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.684211    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:10.684476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:10.684288    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:11.082076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.082036    5098 scope.go:122] "RemoveContainer" containerID="2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"
Oct 31 03:30:11.082565 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:11.082481    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf_odh-ai-gateway-infra(1d89dbe9-76aa-4dd6-91d2-7e02adeb841a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a"
Oct 31 03:30:11.083933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.083904    5098 scope.go:122] "RemoveContainer" containerID="4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"
Oct 31 03:30:11.084298 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:11.084274    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn_odh-ai-gateway-infra(57836d63-4df6-41ff-85a4-3e2aa6768a6f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f"
Oct 31 03:30:11.085755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.085732    5098 scope.go:122] "RemoveContainer" containerID="98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"
Oct 31 03:30:11.086061 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:11.086034    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v_odh-ai-gateway-infra(47b7d5d3-30a5-49f6-a1d0-8be2b89795bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb"
Oct 31 03:30:11.087660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.087601    5098 scope.go:122] "RemoveContainer" containerID="f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"
Oct 31 03:30:11.087986 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:11.087967    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6_odh-ai-gateway-infra(b7684d2d-86c9-4cbc-8c1f-fb6645720deb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb"
Oct 31 03:30:11.089377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.089354    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf" event={"ID":"7add42f2-c873-4393-aa50-a67df59b0276","Type":"ContainerStarted","Data":"d9cc1d08d6a5a30e78b5adbbacf8440959345b70e7f8baa12c24804b7da0bc93"}
Oct 31 03:30:11.089461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.089397    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf" event={"ID":"7add42f2-c873-4393-aa50-a67df59b0276","Type":"ContainerStarted","Data":"c9100c4905ab9aba100e16a2c39c7a226840411efc43d0ea882a3fcc3dfb8c5f"}
Oct 31 03:30:11.118506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.118446    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf" podStartSLOduration=2.118430124 podStartE2EDuration="2.118430124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:11.114959176 +0000 UTC m=+885.640178768" watchObservedRunningTime="2026-10-31 03:30:11.118430124 +0000 UTC m=+885.643649714"
Oct 31 03:30:11.683758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.683718    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:11.683932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:11.683779    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:12.101553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:12.101514    5098 generic.go:356] "Generic (PLEG): container finished" podID="7add42f2-c873-4393-aa50-a67df59b0276" containerID="d9cc1d08d6a5a30e78b5adbbacf8440959345b70e7f8baa12c24804b7da0bc93" exitCode=0
Oct 31 03:30:12.102025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:12.101596    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf" event={"ID":"7add42f2-c873-4393-aa50-a67df59b0276","Type":"ContainerDied","Data":"d9cc1d08d6a5a30e78b5adbbacf8440959345b70e7f8baa12c24804b7da0bc93"}
Oct 31 03:30:12.683924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:12.683876    5098 patch_prober.go:29] interesting pod/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Oct 31 03:30:12.684112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:12.683942    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Oct 31 03:30:13.323486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.323452    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:13.362760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.362659    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\" (UniqueName: \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\") pod \"7add42f2-c873-4393-aa50-a67df59b0276\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") "
Oct 31 03:30:13.362760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.362728    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\") pod \"7add42f2-c873-4393-aa50-a67df59b0276\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") "
Oct 31 03:30:13.362999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.362773    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\") pod \"7add42f2-c873-4393-aa50-a67df59b0276\" (UID: \"7add42f2-c873-4393-aa50-a67df59b0276\") "
Oct 31 03:30:13.363474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.363446    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca" pod "7add42f2-c873-4393-aa50-a67df59b0276" (UID: "7add42f2-c873-4393-aa50-a67df59b0276"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:13.367202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.367147    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca" pod "7add42f2-c873-4393-aa50-a67df59b0276" (UID: "7add42f2-c873-4393-aa50-a67df59b0276"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:13.367581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.367555    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q" pod "7add42f2-c873-4393-aa50-a67df59b0276" (UID: "7add42f2-c873-4393-aa50-a67df59b0276"). InnerVolumeSpecName "kube-api-access-pkd6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:13.465630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.465244    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkd6q\" (UniqueName: \"kubernetes.io/projected/7add42f2-c873-4393-aa50-a67df59b0276-kube-api-access-pkd6q\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:13.465630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.465286    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:13.465630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:13.465301    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7add42f2-c873-4393-aa50-a67df59b0276-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:14.099757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.099720    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:14.101441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.101345    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:14.103220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.103197    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:30:14.189627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.187367    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"
Oct 31 03:30:14.189627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.188903    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf" event={"ID":"7add42f2-c873-4393-aa50-a67df59b0276","Type":"ContainerDied","Data":"c9100c4905ab9aba100e16a2c39c7a226840411efc43d0ea882a3fcc3dfb8c5f"}
Oct 31 03:30:14.189627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:14.188968    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9100c4905ab9aba100e16a2c39c7a226840411efc43d0ea882a3fcc3dfb8c5f"
Oct 31 03:30:17.146316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.146277    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"]
Oct 31 03:30:17.147395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.147366    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7add42f2-c873-4393-aa50-a67df59b0276" containerName="revoke-keys"
Oct 31 03:30:17.147395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.147393    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7add42f2-c873-4393-aa50-a67df59b0276" containerName="revoke-keys"
Oct 31 03:30:17.148083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.148059    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7add42f2-c873-4393-aa50-a67df59b0276" containerName="revoke-keys"
Oct 31 03:30:17.152805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.152780    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.155725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.155696    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"]
Oct 31 03:30:17.156814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.156789    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:30:17.296752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.296706    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgn9c\" (UniqueName: \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.296937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.296760    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.296995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.296971    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.401081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.398710    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.401081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.398933    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.401081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.399197    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgn9c\" (UniqueName: \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.401081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.400442    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.412924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.412879    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.414716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.414126    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgn9c\" (UniqueName: \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\") pod \"maas-api-revoke-sub-b4761b44-9vmgt\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.476405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.476362    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:17.762857 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:17.762825    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e676b_f70d_47fb_8905_e35730f65faf.slice/crio-1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc WatchSource:0}: Error finding container 1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc: Status 404 returned error can't find the container with id 1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc
Oct 31 03:30:17.764933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:17.764901    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"]
Oct 31 03:30:18.222752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:18.222700    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt" event={"ID":"e90e676b-f70d-47fb-8905-e35730f65faf","Type":"ContainerStarted","Data":"1edf2b3932ef5692e6cf7c17ddf4892a5dac55306c8d1651361b986cb5df0601"}
Oct 31 03:30:18.223130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:18.222776    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt" event={"ID":"e90e676b-f70d-47fb-8905-e35730f65faf","Type":"ContainerStarted","Data":"1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc"}
Oct 31 03:30:18.239476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:18.239424    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt" podStartSLOduration=1.2394048149999999 podStartE2EDuration="1.239404815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:18.238055088 +0000 UTC m=+892.763274677" watchObservedRunningTime="2026-10-31 03:30:18.239404815 +0000 UTC m=+892.764624403"
Oct 31 03:30:19.185184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.185143    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:30:19.190108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.190080    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.289667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.288553    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:30:19.295716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.295683    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:30:19.298136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.298104    5098 generic.go:356] "Generic (PLEG): container finished" podID="e90e676b-f70d-47fb-8905-e35730f65faf" containerID="1edf2b3932ef5692e6cf7c17ddf4892a5dac55306c8d1651361b986cb5df0601" exitCode=0
Oct 31 03:30:19.307901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.307859    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt" event={"ID":"e90e676b-f70d-47fb-8905-e35730f65faf","Type":"ContainerDied","Data":"1edf2b3932ef5692e6cf7c17ddf4892a5dac55306c8d1651361b986cb5df0601"}
Oct 31 03:30:19.308089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.307969    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.315507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.315441    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:30:19.353873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.353831    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.353977    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354081    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354250    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9dnt\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354307    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354449    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354501    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354619    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.354695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.354660    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.456491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456381    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.456491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456439    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.456737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456502    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.456737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.456737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456585    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.456737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456617    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.456912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456778    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.456912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456888    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.457003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.456975    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v97sr\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.457059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457014    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9dnt\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457070    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457154    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457194    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457206    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457239    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.457385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457322    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457374    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457446    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.457509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457451    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457507    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457597    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.457660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.457628    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.458197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.458175    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.459703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.459679    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.459884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.459868    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.464780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.464756    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.464865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.464761    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9dnt\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.545372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.544814    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:30:19.551742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.551707    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.558809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.558766    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v97sr\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.558956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.558918    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.559030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.559001    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.560325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560282    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560748    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560790    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560821    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560845    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.560876    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.561043    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.562098    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.562370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.562371    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.565090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.562642    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.565090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.562894    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.569176    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.572555    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.573917    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.579210    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.580841    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v97sr\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\") pod \"e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") " pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.585632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.581488    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:19.608428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.604985    5098 scope.go:122] "RemoveContainer" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"
Oct 31 03:30:19.627751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.627723    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:19.662985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.662904    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6lf9\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663060    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663103    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663203    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663413    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663474    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663524    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663734    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.663939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.663814    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.765355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.765315    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.765554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.765530    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.765645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.765582    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.765714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.765655    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.766050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766027    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.766157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766062    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.766157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766072    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.766282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766238    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6lf9\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766351    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766375    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766385    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.766421    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.767059    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.767319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.767194    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.849587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.849556    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.849900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.849875    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.850018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.849946    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.850112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.850089    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6lf9\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\") pod \"e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") " pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.924593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.924547    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:19.924773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.924584    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:19.924773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.924697    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:19.924773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.924746    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:19.929003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.928841    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:30:19.944994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.944968    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:30:19.947163 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:19.947135    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c6bb59_428a_4433_9362_437e10c1c24f.slice/crio-fc8af709e78960b2a59725cb5f22a3900df0f411e345635940f097863c192199 WatchSource:0}: Error finding container fc8af709e78960b2a59725cb5f22a3900df0f411e345635940f097863c192199: Status 404 returned error can't find the container with id fc8af709e78960b2a59725cb5f22a3900df0f411e345635940f097863c192199
Oct 31 03:30:19.954212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.954180    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:19.954332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.954246    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:19.954332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:19.954275    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:20.329261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.329125    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" event={"ID":"f6c6bb59-428a-4433-9362-437e10c1c24f","Type":"ContainerStarted","Data":"c8e4538a3de963d6f988a359e9131ea28aad1c2423d040b3dbc84006c8da9051"}
Oct 31 03:30:20.329261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.329223    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" event={"ID":"f6c6bb59-428a-4433-9362-437e10c1c24f","Type":"ContainerStarted","Data":"fc8af709e78960b2a59725cb5f22a3900df0f411e345635940f097863c192199"}
Oct 31 03:30:20.332234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.332138    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerStarted","Data":"e11c4ae833ee54557e5b7dd4c5d0ea502dccc1e4a566cce4d615f796962bdb82"}
Oct 31 03:30:20.334133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.334102    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" event={"ID":"9331923e-a2df-4258-b289-9970c6cd3eda","Type":"ContainerStarted","Data":"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"}
Oct 31 03:30:20.334265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.334156    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" event={"ID":"9331923e-a2df-4258-b289-9970c6cd3eda","Type":"ContainerStarted","Data":"fcdfc81d48adc9d0407fa8f0c35d5a75091ac9ad361271b4149da05c95e79903"}
Oct 31 03:30:20.393123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.393058    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" podStartSLOduration=20.393038771 podStartE2EDuration="20.393038771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:20.387026674 +0000 UTC m=+894.912246265" watchObservedRunningTime="2026-10-31 03:30:20.393038771 +0000 UTC m=+894.918258362"
Oct 31 03:30:20.393466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.393433    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podStartSLOduration=1.393423798 podStartE2EDuration="1.393423798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:20.367976259 +0000 UTC m=+894.893195847" watchObservedRunningTime="2026-10-31 03:30:20.393423798 +0000 UTC m=+894.918643392"
Oct 31 03:30:20.424156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.423951    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:30:20.429252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.429187    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podStartSLOduration=1.4291661310000001 podStartE2EDuration="1.429166131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:20.409828581 +0000 UTC m=+894.935048171" watchObservedRunningTime="2026-10-31 03:30:20.429166131 +0000 UTC m=+894.954385721"
Oct 31 03:30:20.432487 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:20.432441    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d5fdd6_21f3_4859_a497_8488c0163461.slice/crio-01780d88f3f3d3bd7ec532fc156213918e0025a7b2567da49835d92e8e04f027 WatchSource:0}: Error finding container 01780d88f3f3d3bd7ec532fc156213918e0025a7b2567da49835d92e8e04f027: Status 404 returned error can't find the container with id 01780d88f3f3d3bd7ec532fc156213918e0025a7b2567da49835d92e8e04f027
Oct 31 03:30:20.438033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.436934    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:20.438033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.437026    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:20.438033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.437067    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:30:20.581949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.581920    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:20.593006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.592972    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:20.604132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.604104    5098 scope.go:122] "RemoveContainer" containerID="4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"
Oct 31 03:30:20.629207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.629183    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:20.708035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.706819    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\" (UniqueName: \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\") pod \"e90e676b-f70d-47fb-8905-e35730f65faf\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") "
Oct 31 03:30:20.708035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.707006    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\") pod \"e90e676b-f70d-47fb-8905-e35730f65faf\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") "
Oct 31 03:30:20.708035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.707070    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\") pod \"e90e676b-f70d-47fb-8905-e35730f65faf\" (UID: \"e90e676b-f70d-47fb-8905-e35730f65faf\") "
Oct 31 03:30:20.708035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.707574    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca" pod "e90e676b-f70d-47fb-8905-e35730f65faf" (UID: "e90e676b-f70d-47fb-8905-e35730f65faf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:20.708624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.708590    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:20.712096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.712058    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca" pod "e90e676b-f70d-47fb-8905-e35730f65faf" (UID: "e90e676b-f70d-47fb-8905-e35730f65faf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:20.713912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.713880    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c" pod "e90e676b-f70d-47fb-8905-e35730f65faf" (UID: "e90e676b-f70d-47fb-8905-e35730f65faf"). InnerVolumeSpecName "kube-api-access-pgn9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:20.811052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.811010    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgn9c\" (UniqueName: \"kubernetes.io/projected/e90e676b-f70d-47fb-8905-e35730f65faf-kube-api-access-pgn9c\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:20.811052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:20.811057    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e676b-f70d-47fb-8905-e35730f65faf-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:21.345207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.345170    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerStarted","Data":"2d163dc3e1d3a20e3f06ecdf39de492a7470167e339d30f0f9ee402fdcd5473e"}
Oct 31 03:30:21.346723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.346697    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"
Oct 31 03:30:21.346860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.346745    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt" event={"ID":"e90e676b-f70d-47fb-8905-e35730f65faf","Type":"ContainerDied","Data":"1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc"}
Oct 31 03:30:21.346860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.346793    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b1a4ee918aaa785d52669583c5176bb61f7a42174dffbcb689e71814ea47cbc"
Oct 31 03:30:21.348625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.348575    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" event={"ID":"63d5fdd6-21f3-4859-a497-8488c0163461","Type":"ContainerStarted","Data":"2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26"}
Oct 31 03:30:21.348758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.348698    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" event={"ID":"63d5fdd6-21f3-4859-a497-8488c0163461","Type":"ContainerStarted","Data":"01780d88f3f3d3bd7ec532fc156213918e0025a7b2567da49835d92e8e04f027"}
Oct 31 03:30:21.409229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.409169    5098 generic.go:356] "Generic (PLEG): container finished" podID="98a7637d-8306-4f42-beb4-6452b1174e78" containerID="e11c4ae833ee54557e5b7dd4c5d0ea502dccc1e4a566cce4d615f796962bdb82" exitCode=60
Oct 31 03:30:21.410261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.410228    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerDied","Data":"e11c4ae833ee54557e5b7dd4c5d0ea502dccc1e4a566cce4d615f796962bdb82"}
Oct 31 03:30:21.410399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.410290    5098 scope.go:122] "RemoveContainer" containerID="38c9f5ef52030e7e577912ecb8dd6e3bfd855ac06fae85a675b1d74026ee4ac2"
Oct 31 03:30:21.411410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.411389    5098 scope.go:122] "RemoveContainer" containerID="e11c4ae833ee54557e5b7dd4c5d0ea502dccc1e4a566cce4d615f796962bdb82"
Oct 31 03:30:21.411737 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:21.411711    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685_odh-ai-gateway-infra(98a7637d-8306-4f42-beb4-6452b1174e78)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" podUID="98a7637d-8306-4f42-beb4-6452b1174e78"
Oct 31 03:30:21.426302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.426270    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:21.437225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.437178    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podStartSLOduration=2.437165416 podStartE2EDuration="2.437165416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:21.433030657 +0000 UTC m=+895.958250247" watchObservedRunningTime="2026-10-31 03:30:21.437165416 +0000 UTC m=+895.962385004"
Oct 31 03:30:21.604774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.604672    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:21.604774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.604740    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:21.620594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.620525    5098 scope.go:122] "RemoveContainer" containerID="f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"
Oct 31 03:30:21.630396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.630360    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:21.630521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.630408    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:21.927763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.927664    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:21.931839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.931804    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:21.932002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:21.931877    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:22.433761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.432271    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:22.628632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.603368    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:22.628632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.603452    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:22.628632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.605007    5098 scope.go:122] "RemoveContainer" containerID="98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"
Oct 31 03:30:22.635429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.635385    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:22.635589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.635462    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:22.646974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.646127    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerStarted","Data":"1a0a28e8144b63e01c88bfc3e544891d97f8a04421331a4958022c2838783164"}
Oct 31 03:30:22.739021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.738977    5098 generic.go:356] "Generic (PLEG): container finished" podID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerID="2d163dc3e1d3a20e3f06ecdf39de492a7470167e339d30f0f9ee402fdcd5473e" exitCode=60
Oct 31 03:30:22.739021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.739011    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerDied","Data":"2d163dc3e1d3a20e3f06ecdf39de492a7470167e339d30f0f9ee402fdcd5473e"}
Oct 31 03:30:22.739256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.739075    5098 scope.go:122] "RemoveContainer" containerID="4682145741fa4aebd3a301aabf20ff26c98d88cda5226fcab1dc4649e93cb23e"
Oct 31 03:30:22.938883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.934095    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:22.938883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:22.934150    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:23.658743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.640779    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"]
Oct 31 03:30:23.658743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.641939    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e90e676b-f70d-47fb-8905-e35730f65faf" containerName="revoke-keys"
Oct 31 03:30:23.658743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.641966    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e676b-f70d-47fb-8905-e35730f65faf" containerName="revoke-keys"
Oct 31 03:30:23.658743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.642217    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90e676b-f70d-47fb-8905-e35730f65faf" containerName="revoke-keys"
Oct 31 03:30:23.701649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.667319    5098 scope.go:122] "RemoveContainer" containerID="2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"
Oct 31 03:30:23.993411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.970254    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:23.993411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:23.991925    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.108996    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.109058    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.121370    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.121443    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.144473    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.144577    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.148880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.144667    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slftw\" (UniqueName: \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.374633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.367040    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.374633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.367100    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:24.457453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.452496    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn" event={"ID":"57836d63-4df6-41ff-85a4-3e2aa6768a6f","Type":"ContainerDied","Data":"37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863"}
Oct 31 03:30:24.457453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.452572    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cca9737aebf550b3e2e8bd732c91760a3aa2a1f7218ae5faf75c53eb11f863"
Oct 31 03:30:24.457453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.452592    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"]
Oct 31 03:30:24.457453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.452650    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:24.614282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.543394    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.614282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.543522    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.614282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.543626    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slftw\" (UniqueName: \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.636566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.636473    5098 generic.go:356] "Generic (PLEG): container finished" podID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerID="1a0a28e8144b63e01c88bfc3e544891d97f8a04421331a4958022c2838783164" exitCode=60
Oct 31 03:30:24.637001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.636972    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6" event={"ID":"b7684d2d-86c9-4cbc-8c1f-fb6645720deb","Type":"ContainerDied","Data":"1a0a28e8144b63e01c88bfc3e544891d97f8a04421331a4958022c2838783164"}
Oct 31 03:30:24.637851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.637825    5098 scope.go:122] "RemoveContainer" containerID="f41c6af1ea0361c83e3ca5db59af2a3809e554fc588bacdf934c14bbb3ca5600"
Oct 31 03:30:24.641276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.641244    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:24.648639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.648232    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.648639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.648281    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:24.710348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.654956    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:24.710348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.709814    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:24.710348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.710244    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:24.792283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.773688    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:24.834925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.813227    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685" event={"ID":"98a7637d-8306-4f42-beb4-6452b1174e78","Type":"ContainerDied","Data":"8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933"}
Oct 31 03:30:24.834925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:24.813298    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933"
Oct 31 03:30:25.048632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.026951    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\") pod \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") "
Oct 31 03:30:25.048632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.027129    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") pod \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") "
Oct 31 03:30:25.052631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.051516    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:25.155592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.102640    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:25.155592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.102701    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:25.204557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.183270    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:25.204557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.204413    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slftw\" (UniqueName: \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:25.542726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.458915    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\" (UniqueName: \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\") pod \"98a7637d-8306-4f42-beb4-6452b1174e78\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") "
Oct 31 03:30:25.542726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.459012    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\") pod \"98a7637d-8306-4f42-beb4-6452b1174e78\" (UID: \"98a7637d-8306-4f42-beb4-6452b1174e78\") "
Oct 31 03:30:25.542726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.467278    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:25.542726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.515951    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca" pod "98a7637d-8306-4f42-beb4-6452b1174e78" (UID: "98a7637d-8306-4f42-beb4-6452b1174e78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:25.542726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.516154    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb" pod "98a7637d-8306-4f42-beb4-6452b1174e78" (UID: "98a7637d-8306-4f42-beb4-6452b1174e78"). InnerVolumeSpecName "kube-api-access-gwbdb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.631597    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwbdb\" (UniqueName: \"kubernetes.io/projected/98a7637d-8306-4f42-beb4-6452b1174e78-kube-api-access-gwbdb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.631653    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98a7637d-8306-4f42-beb4-6452b1174e78-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.635096    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.648363    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.736321    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca" pod "57836d63-4df6-41ff-85a4-3e2aa6768a6f" (UID: "57836d63-4df6-41ff-85a4-3e2aa6768a6f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.738140    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.738193    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.738343    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.738385    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.747466    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx" pod "57836d63-4df6-41ff-85a4-3e2aa6768a6f" (UID: "57836d63-4df6-41ff-85a4-3e2aa6768a6f"). InnerVolumeSpecName "kube-api-access-ngckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:25.881429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.765418    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d94a2437-6vnh4\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:26.036706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.984190    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:26.036706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:25.984248    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:26.535634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.508064    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") pod \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\" (UID: \"57836d63-4df6-41ff-85a4-3e2aa6768a6f\") "
Oct 31 03:30:26.535634 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:26.509647    5098 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/57836d63-4df6-41ff-85a4-3e2aa6768a6f/volumes/kubernetes.io~projected/kube-api-access-ngckx
Oct 31 03:30:26.535634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.509731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx" pod "57836d63-4df6-41ff-85a4-3e2aa6768a6f" (UID: "57836d63-4df6-41ff-85a4-3e2aa6768a6f"). InnerVolumeSpecName "kube-api-access-ngckx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:26.691873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.665324    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:26.727631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.693992    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57836d63-4df6-41ff-85a4-3e2aa6768a6f-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:26.732931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.732875    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:26.734868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.734823    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:26.847787 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:26.847289    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.204s"
Oct 31 03:30:26.859644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.847990    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngckx\" (UniqueName: \"kubernetes.io/projected/57836d63-4df6-41ff-85a4-3e2aa6768a6f-kube-api-access-ngckx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:26.938869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.938757    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:26.949714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:26.939776    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:27.093710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.015250    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:27.141383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.140959    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:27.141383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.034495    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:27.185708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.183874    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:27.229574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.229523    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerStarted","Data":"8b364b41006e8f473bd6bae453fae3d771fc5c5b3e4e8b67a538961132433a30"}
Oct 31 03:30:27.239153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.239120    5098 generic.go:356] "Generic (PLEG): container finished" podID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerID="a4b8d946fa88bc5c0a4a02ee5e27e218f6f36efae223f09592659181a29fa9be" exitCode=60
Oct 31 03:30:27.239553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.239525    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerDied","Data":"a4b8d946fa88bc5c0a4a02ee5e27e218f6f36efae223f09592659181a29fa9be"}
Oct 31 03:30:27.239755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.239737    5098 scope.go:122] "RemoveContainer" containerID="98fad448794d616ab3cade8cc9604b8526c58e974c7093964236a718427d2439"
Oct 31 03:30:27.240375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.240355    5098 scope.go:122] "RemoveContainer" containerID="a4b8d946fa88bc5c0a4a02ee5e27e218f6f36efae223f09592659181a29fa9be"
Oct 31 03:30:27.247287 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:27.247237    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v_odh-ai-gateway-infra(47b7d5d3-30a5-49f6-a1d0-8be2b89795bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb"
Oct 31 03:30:27.247784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.247761    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"
Oct 31 03:30:27.247968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.247948    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"
Oct 31 03:30:27.248470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.248056    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"
Oct 31 03:30:27.295942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.294346    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\" (UniqueName: \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\") pod \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") "
Oct 31 03:30:27.295942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.294410    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\") pod \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\" (UID: \"b7684d2d-86c9-4cbc-8c1f-fb6645720deb\") "
Oct 31 03:30:27.295942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.295601    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca" pod "b7684d2d-86c9-4cbc-8c1f-fb6645720deb" (UID: "b7684d2d-86c9-4cbc-8c1f-fb6645720deb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:27.299857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.299817    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6" pod "b7684d2d-86c9-4cbc-8c1f-fb6645720deb" (UID: "b7684d2d-86c9-4cbc-8c1f-fb6645720deb"). InnerVolumeSpecName "kube-api-access-s8nc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:27.336347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.336309    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:27.368027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.367949    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890290-4s685"]
Oct 31 03:30:27.409717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.409581    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8nc6\" (UniqueName: \"kubernetes.io/projected/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-kube-api-access-s8nc6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:27.409948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.409930    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7684d2d-86c9-4cbc-8c1f-fb6645720deb-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:27.420591 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:27.420555    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182f0f6f_4678_4264_a943_c4aca911b925.slice/crio-54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3 WatchSource:0}: Error finding container 54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3: Status 404 returned error can't find the container with id 54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3
Oct 31 03:30:27.423649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.423459    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:27.432055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.432024    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f20dc4-29890290-9f2sn"]
Oct 31 03:30:27.442219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.441131    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"]
Oct 31 03:30:27.518267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.518235    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:30:27.521139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.521095    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:30:27.583636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.582128    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused" start-of-body=
Oct 31 03:30:27.583636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.582201    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": dial tcp 10.132.0.120:15021: connect: connection refused"
Oct 31 03:30:27.621330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.620255    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" path="/var/lib/kubelet/pods/57836d63-4df6-41ff-85a4-3e2aa6768a6f/volumes"
Oct 31 03:30:27.623889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.622197    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" path="/var/lib/kubelet/pods/98a7637d-8306-4f42-beb4-6452b1174e78/volumes"
Oct 31 03:30:27.623889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.622819    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:27.627835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.627759    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-6c6050-29890290-h6hc6"]
Oct 31 03:30:27.629701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.629671    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body=
Oct 31 03:30:27.629808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.629740    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused"
Oct 31 03:30:27.925357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.925323    5098 patch_prober.go:29] interesting pod/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 03:30:27.925528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:27.925383    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 03:30:28.273371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.271988    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4" event={"ID":"182f0f6f-4678-4264-a943-c4aca911b925","Type":"ContainerStarted","Data":"460d6c6648091ef234cdff8df94d4e8f8aca146837b6161861de6f822b00eec7"}
Oct 31 03:30:28.273371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.272063    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4" event={"ID":"182f0f6f-4678-4264-a943-c4aca911b925","Type":"ContainerStarted","Data":"54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3"}
Oct 31 03:30:28.276248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.276112    5098 scope.go:122] "RemoveContainer" containerID="1a0a28e8144b63e01c88bfc3e544891d97f8a04421331a4958022c2838783164"
Oct 31 03:30:28.281532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.281418    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:28.281715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.281693    5098 generic.go:356] "Generic (PLEG): container finished" podID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerID="8b364b41006e8f473bd6bae453fae3d771fc5c5b3e4e8b67a538961132433a30" exitCode=60
Oct 31 03:30:28.281915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.281813    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerDied","Data":"8b364b41006e8f473bd6bae453fae3d771fc5c5b3e4e8b67a538961132433a30"}
Oct 31 03:30:28.297480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.297417    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4" podStartSLOduration=5.297397119 podStartE2EDuration="5.297397119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:28.293456282 +0000 UTC m=+902.818675871" watchObservedRunningTime="2026-10-31 03:30:28.297397119 +0000 UTC m=+902.822616709"
Oct 31 03:30:28.313191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.312400    5098 scope.go:122] "RemoveContainer" containerID="2aaa15268f3a526487a97f785d4688fb86cb53467894039a661ecf133127313e"
Oct 31 03:30:28.333013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.332976    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:28.776668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.775375    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:28.786501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.786469    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:28.873665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.873531    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:28.876357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.876323    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\" (UniqueName: \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\") pod \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") "
Oct 31 03:30:28.876513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.876375    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\") pod \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") "
Oct 31 03:30:28.876708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.876687    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\" (UniqueName: \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\") pod \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\" (UID: \"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a\") "
Oct 31 03:30:28.876820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.876752    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\") pod \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\" (UID: \"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb\") "
Oct 31 03:30:28.878335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.877983    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca" pod "1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" (UID: "1d89dbe9-76aa-4dd6-91d2-7e02adeb841a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:28.878899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.878714    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca" pod "47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" (UID: "47b7d5d3-30a5-49f6-a1d0-8be2b89795bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:28.880938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.880917    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:28.894666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.889812    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5" pod "1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" (UID: "1d89dbe9-76aa-4dd6-91d2-7e02adeb841a"). InnerVolumeSpecName "kube-api-access-pnjk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:28.894666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.889931    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66" pod "47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" (UID: "47b7d5d3-30a5-49f6-a1d0-8be2b89795bb"). InnerVolumeSpecName "kube-api-access-rjd66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:28.894666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.891221    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:30:28.980514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.979553    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnjk5\" (UniqueName: \"kubernetes.io/projected/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-kube-api-access-pnjk5\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:28.980514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.979590    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:28.980514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.979626    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjd66\" (UniqueName: \"kubernetes.io/projected/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb-kube-api-access-rjd66\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:28.980514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:28.979642    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:29.063509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.063436    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:29.065601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.065573    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:29.068359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.068329    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:30:29.354626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.354571    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v" event={"ID":"47b7d5d3-30a5-49f6-a1d0-8be2b89795bb","Type":"ContainerDied","Data":"c4d60a286ff23e97ff6b872c47d7d031f388e2d0b755ed3640a17be6679cb754"}
Oct 31 03:30:29.355140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.354783    5098 scope.go:122] "RemoveContainer" containerID="a4b8d946fa88bc5c0a4a02ee5e27e218f6f36efae223f09592659181a29fa9be"
Oct 31 03:30:29.355140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.354852    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"
Oct 31 03:30:29.360406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.360370    5098 generic.go:356] "Generic (PLEG): container finished" podID="182f0f6f-4678-4264-a943-c4aca911b925" containerID="460d6c6648091ef234cdff8df94d4e8f8aca146837b6161861de6f822b00eec7" exitCode=0
Oct 31 03:30:29.360545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.360416    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4" event={"ID":"182f0f6f-4678-4264-a943-c4aca911b925","Type":"ContainerDied","Data":"460d6c6648091ef234cdff8df94d4e8f8aca146837b6161861de6f822b00eec7"}
Oct 31 03:30:29.367911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.364686    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf" event={"ID":"1d89dbe9-76aa-4dd6-91d2-7e02adeb841a","Type":"ContainerDied","Data":"d366c404ce3d6b92167562368f3115e1591ea7dd6f4a4cd32a34c5b1f4737a8e"}
Oct 31 03:30:29.367911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.364722    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"
Oct 31 03:30:29.392423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.384024    5098 scope.go:122] "RemoveContainer" containerID="8b364b41006e8f473bd6bae453fae3d771fc5c5b3e4e8b67a538961132433a30"
Oct 31 03:30:29.478973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.478937    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:29.486233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.486131    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-090ebd-29890290-ng7mf"]
Oct 31 03:30:29.493211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.493173    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:29.507004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.506965    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890290-d7k4v"]
Oct 31 03:30:29.535789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.535751    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:29.536354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.536330    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:29.537706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.537678    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:30:29.620894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.620805    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" path="/var/lib/kubelet/pods/1d89dbe9-76aa-4dd6-91d2-7e02adeb841a/volumes"
Oct 31 03:30:29.621596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.621566    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" path="/var/lib/kubelet/pods/47b7d5d3-30a5-49f6-a1d0-8be2b89795bb/volumes"
Oct 31 03:30:29.622258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:29.622236    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" path="/var/lib/kubelet/pods/b7684d2d-86c9-4cbc-8c1f-fb6645720deb/volumes"
Oct 31 03:30:30.113474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.113443    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"]
Oct 31 03:30:30.114862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.114774    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.115092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115074    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.115218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115206    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.115323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115310    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.115419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115407    5098 cpu_manager.go:414] "containerMap: removing container" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.115497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115485    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.115583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115574    5098 cpu_manager.go:414] "containerMap: removing container" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.115673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115662    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.115750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115740    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.115830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115819    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.115907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115897    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.115975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.115965    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.116081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116071    5098 cpu_manager.go:414] "containerMap: removing container" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.116157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116147    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.116234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116226    5098 cpu_manager.go:414] "containerMap: removing container" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.116304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116293    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.116385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116376    5098 cpu_manager.go:414] "containerMap: removing container" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.116453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116443    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.116580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116569    5098 cpu_manager.go:414] "containerMap: removing container" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.116674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116662    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.116934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.116917    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.117027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117017    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.117117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117107    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.117196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117188    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.117280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117270    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.117351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117343    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.117426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117417    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:30.117502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117493    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:30.117576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117567    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:30.117672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117662    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:30.117744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117735    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.117842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.117820    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:30.132897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.129945    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.132897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.129835    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"]
Oct 31 03:30:30.224314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.224274    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgtgn\" (UniqueName: \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.224503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.224411    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.224664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.224644    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.326901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.326860    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.327074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.326977    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgtgn\" (UniqueName: \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.327074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.327045    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.328849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.328334    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.332317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.332289    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.337653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.337627    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgtgn\" (UniqueName: \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\") pod \"maas-api-revoke-sub-acfa9078-9zvrl\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.472040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.472007    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:30.622228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.622152    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:30.736302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.736197    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\" (UniqueName: \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\") pod \"182f0f6f-4678-4264-a943-c4aca911b925\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") "
Oct 31 03:30:30.736470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.736369    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\") pod \"182f0f6f-4678-4264-a943-c4aca911b925\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") "
Oct 31 03:30:30.736470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.736415    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\") pod \"182f0f6f-4678-4264-a943-c4aca911b925\" (UID: \"182f0f6f-4678-4264-a943-c4aca911b925\") "
Oct 31 03:30:30.736988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.736957    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca" pod "182f0f6f-4678-4264-a943-c4aca911b925" (UID: "182f0f6f-4678-4264-a943-c4aca911b925"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:30.737438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.737409    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:30.739665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.739637    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca" pod "182f0f6f-4678-4264-a943-c4aca911b925" (UID: "182f0f6f-4678-4264-a943-c4aca911b925"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:30.739765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.739686    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw" pod "182f0f6f-4678-4264-a943-c4aca911b925" (UID: "182f0f6f-4678-4264-a943-c4aca911b925"). InnerVolumeSpecName "kube-api-access-slftw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:30.797744 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:30.797711    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead68c66_d9c9_4e86_bbd4_83cde2e7ce43.slice/crio-f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4 WatchSource:0}: Error finding container f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4: Status 404 returned error can't find the container with id f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4
Oct 31 03:30:30.799255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.799228    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"]
Oct 31 03:30:30.844138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.844097    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slftw\" (UniqueName: \"kubernetes.io/projected/182f0f6f-4678-4264-a943-c4aca911b925-kube-api-access-slftw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:30.844335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:30.844144    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/182f0f6f-4678-4264-a943-c4aca911b925-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:31.061205 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:31.061164    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:30:31.392494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.392408    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"
Oct 31 03:30:31.392494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.392439    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4" event={"ID":"182f0f6f-4678-4264-a943-c4aca911b925","Type":"ContainerDied","Data":"54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3"}
Oct 31 03:30:31.392717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.392495    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54f7be7e93cc2b9cc13c6e3b450092663390199c5c8ede3f084ec8436308cfb3"
Oct 31 03:30:31.394622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.394576    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl" event={"ID":"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43","Type":"ContainerStarted","Data":"d0c4ea776e44a0051b5cc0fa53bb95111d569b8faa10d86149d8aa82bcbdb326"}
Oct 31 03:30:31.394728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.394643    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl" event={"ID":"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43","Type":"ContainerStarted","Data":"f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4"}
Oct 31 03:30:31.426747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.426689    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl" podStartSLOduration=1.426671035 podStartE2EDuration="1.426671035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:31.422773443 +0000 UTC m=+905.947993055" watchObservedRunningTime="2026-10-31 03:30:31.426671035 +0000 UTC m=+905.951890650"
Oct 31 03:30:31.879896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.879857    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"]
Oct 31 03:30:31.880809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880789    5098 cpu_manager.go:414] "containerMap: removing container" podUID="182f0f6f-4678-4264-a943-c4aca911b925" containerName="revoke-keys"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880811    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="182f0f6f-4678-4264-a943-c4aca911b925" containerName="revoke-keys"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880832    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880837    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d89dbe9-76aa-4dd6-91d2-7e02adeb841a" containerName="cleanup"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880848    5098 cpu_manager.go:414] "containerMap: removing container" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880854    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880864    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:31.880878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880869    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880885    5098 cpu_manager.go:414] "containerMap: removing container" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880890    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880903    5098 cpu_manager.go:414] "containerMap: removing container" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.880908    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57836d63-4df6-41ff-85a4-3e2aa6768a6f" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.881009    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="182f0f6f-4678-4264-a943-c4aca911b925" containerName="revoke-keys"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.881019    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="98a7637d-8306-4f42-beb4-6452b1174e78" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.881027    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b7d5d3-30a5-49f6-a1d0-8be2b89795bb" containerName="cleanup"
Oct 31 03:30:31.881126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.881033    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7684d2d-86c9-4cbc-8c1f-fb6645720deb" containerName="cleanup"
Oct 31 03:30:31.885002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.884980    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:31.893305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.893273    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"]
Oct 31 03:30:31.969162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.969112    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2nd8\" (UniqueName: \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:31.969378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.969359    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:31.969617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:31.969583    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.070906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.070872    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.071072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.071027    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.071178 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.071070    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2nd8\" (UniqueName: \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.071768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.071744    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.073311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.073293    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.080100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.080076    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2nd8\" (UniqueName: \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\") pod \"maas-api-revoke-sub-d2dc3b72-7wrrk\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.205386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.205353    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:32.405272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.405181    5098 generic.go:356] "Generic (PLEG): container finished" podID="ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" containerID="d0c4ea776e44a0051b5cc0fa53bb95111d569b8faa10d86149d8aa82bcbdb326" exitCode=0
Oct 31 03:30:32.405272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.405255    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl" event={"ID":"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43","Type":"ContainerDied","Data":"d0c4ea776e44a0051b5cc0fa53bb95111d569b8faa10d86149d8aa82bcbdb326"}
Oct 31 03:30:32.499188 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:32.499152    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d979b4_6629_4d8e_b499_c373911fa1e4.slice/crio-c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29 WatchSource:0}: Error finding container c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29: Status 404 returned error can't find the container with id c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29
Oct 31 03:30:32.499924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:32.499894    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"]
Oct 31 03:30:33.468034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.467922    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk" event={"ID":"00d979b4-6629-4d8e-b499-c373911fa1e4","Type":"ContainerStarted","Data":"cf6eeddb79d1afb4931a8ad225e2b99b519c3bf842fc7b2a8c6bda8ba90d81ff"}
Oct 31 03:30:33.468034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.467990    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk" event={"ID":"00d979b4-6629-4d8e-b499-c373911fa1e4","Type":"ContainerStarted","Data":"c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29"}
Oct 31 03:30:33.573758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.573712    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:33.574439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.574404    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:30:33.597635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.585642    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk" podStartSLOduration=2.585600584 podStartE2EDuration="2.585600584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:33.501973144 +0000 UTC m=+908.027192781" watchObservedRunningTime="2026-10-31 03:30:33.585600584 +0000 UTC m=+908.110820174"
Oct 31 03:30:33.693544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.691825    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:33.877633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:33.858196    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.018519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.017720    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.040743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.040695    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.397820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.397714    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:30:34.539633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.532838    5098 generic.go:356] "Generic (PLEG): container finished" podID="00d979b4-6629-4d8e-b499-c373911fa1e4" containerID="cf6eeddb79d1afb4931a8ad225e2b99b519c3bf842fc7b2a8c6bda8ba90d81ff" exitCode=0
Oct 31 03:30:34.539633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.533171    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk" event={"ID":"00d979b4-6629-4d8e-b499-c373911fa1e4","Type":"ContainerDied","Data":"cf6eeddb79d1afb4931a8ad225e2b99b519c3bf842fc7b2a8c6bda8ba90d81ff"}
Oct 31 03:30:34.575750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.575702    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.576504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.576482    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:30:34.577643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.576019    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:30:34.578680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.578662    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:30:34.729737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.725742    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.729737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.726374    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:34.729737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:34.726762    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:30:35.033144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.027563    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.111638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.111373    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.135903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.120832    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:35.135903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.125650    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:30:35.135903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.125737    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:30:35.269823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.269790    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:30:35.388632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.383589    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 31 03:30:35.422091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.417895    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.422091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.418321    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:30:35.497896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.497855    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 03:30:35.582721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.580936    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.652117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.652012    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.751476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.751427    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:35.836453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.775178    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:30:35.836453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:35.835419    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.173787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.173728    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.193434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.185722    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:30:36.453632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.445895    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.506653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.483431    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.669797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.669741    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.676868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.676801    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.924829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.924773    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.999090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:36.980776    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:36.999090 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:36.984206    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.256s"
Oct 31 03:30:37.194222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.193784    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.195628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.194430    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.527703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.526325    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.733595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.733537    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.748259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.748214    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.898689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.895459    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:37.951633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:37.951460    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:38.047687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.025840    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:38.114665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.110667    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:38.311631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.296211    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\") pod \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") "
Oct 31 03:30:38.311631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.296301    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\" (UniqueName: \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\") pod \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") "
Oct 31 03:30:38.311631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.296375    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\") pod \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\" (UID: \"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43\") "
Oct 31 03:30:38.448315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.448253    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:30:38.449897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.449849    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:30:38.500016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.499967    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca" pod "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" (UID: "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:38.570505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.559784    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn" pod "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" (UID: "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43"). InnerVolumeSpecName "kube-api-access-hgtgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:38.584221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.584173    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:38.652634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.633124    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:38.652944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.652909    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgtgn\" (UniqueName: \"kubernetes.io/projected/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-kube-api-access-hgtgn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:38.722634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.704357    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca" pod "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" (UID: "ead68c66-d9c9-4e86-bbd4-83cde2e7ce43"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:38.722924 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:38.704471    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.057s"
Oct 31 03:30:38.723210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.723178    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl" event={"ID":"ead68c66-d9c9-4e86-bbd4-83cde2e7ce43","Type":"ContainerDied","Data":"f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4"}
Oct 31 03:30:38.723361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.723340    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f38fbecf71b27045077687989438ba10de4ac2fb46967d23704cbd516da9e2b4"
Oct 31 03:30:38.764150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.736396    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:38.765451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.759533    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:38.781250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:38.759585    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:39.003771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.003663    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:39.062127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.062062    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:30:39.506634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.502721    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:39.540709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.521572    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:39.591071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.553867    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"
Oct 31 03:30:39.637027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:39.628140    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:30:40.182184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.182143    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:40.580569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.580526    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:40.647801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.582920    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:40.670340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.670310    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:30:40.705048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.703711    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:30:40.706592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.706429    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:30:40.706780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.706750    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:30:40.708461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.708436    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:30:40.716650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.716595    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:40.750432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.750387    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk" event={"ID":"00d979b4-6629-4d8e-b499-c373911fa1e4","Type":"ContainerDied","Data":"c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29"}
Oct 31 03:30:40.753080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.753050    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c05642747dc64c15d676ef783960a558596b1bd6d5567d6c9cf5303801d1bc29"
Oct 31 03:30:40.753348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.753335    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"
Oct 31 03:30:40.754884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.754862    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:30:40.762600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.762564    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:30:40.768099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.768069    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:30:40.770533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.770490    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:30:40.774012 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:40.773982    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead68c66_d9c9_4e86_bbd4_83cde2e7ce43.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:40.808275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.808180    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\") pod \"00d979b4-6629-4d8e-b499-c373911fa1e4\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") "
Oct 31 03:30:40.808418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.808322    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\" (UniqueName: \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\") pod \"00d979b4-6629-4d8e-b499-c373911fa1e4\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") "
Oct 31 03:30:40.808502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.808454    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\") pod \"00d979b4-6629-4d8e-b499-c373911fa1e4\" (UID: \"00d979b4-6629-4d8e-b499-c373911fa1e4\") "
Oct 31 03:30:40.814117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.814075    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca" pod "00d979b4-6629-4d8e-b499-c373911fa1e4" (UID: "00d979b4-6629-4d8e-b499-c373911fa1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:40.818666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.818169    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8" pod "00d979b4-6629-4d8e-b499-c373911fa1e4" (UID: "00d979b4-6629-4d8e-b499-c373911fa1e4"). InnerVolumeSpecName "kube-api-access-d2nd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:40.818854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.818827    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca" pod "00d979b4-6629-4d8e-b499-c373911fa1e4" (UID: "00d979b4-6629-4d8e-b499-c373911fa1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:40.913154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.913111    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2nd8\" (UniqueName: \"kubernetes.io/projected/00d979b4-6629-4d8e-b499-c373911fa1e4-kube-api-access-d2nd8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:40.913154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.913156    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:40.913403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:40.913172    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00d979b4-6629-4d8e-b499-c373911fa1e4-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:41.153130 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:41.153085    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:41.250302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.250231    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"]
Oct 31 03:30:41.251062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251043    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" containerName="revoke-keys"
Oct 31 03:30:41.251062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251065    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" containerName="revoke-keys"
Oct 31 03:30:41.251155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251076    5098 cpu_manager.go:414] "containerMap: removing container" podUID="00d979b4-6629-4d8e-b499-c373911fa1e4" containerName="revoke-keys"
Oct 31 03:30:41.251155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251082    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00d979b4-6629-4d8e-b499-c373911fa1e4" containerName="revoke-keys"
Oct 31 03:30:41.251226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251189    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" containerName="revoke-keys"
Oct 31 03:30:41.251226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.251198    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="00d979b4-6629-4d8e-b499-c373911fa1e4" containerName="revoke-keys"
Oct 31 03:30:41.255504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.255473    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.260254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.260205    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:30:41.260511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.260492    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:30:41.260592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.260561    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:30:41.260592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.260571    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:30:41.263589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.263555    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"]
Oct 31 03:30:41.318625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.318521    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lddht\" (UniqueName: \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.318805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.318687    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.318926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.318903    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.422291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.421685    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lddht\" (UniqueName: \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.424501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.424465    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.425912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.425882    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.427381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.426967    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.460286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.460184    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.478151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.470569    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lddht\" (UniqueName: \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\") pod \"maas-api-revoke-sub-71058046-mxgxh\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.574081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.573992    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:41.867729 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:41.867693    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4d79bea_3f93_462a_9255_460b4e0a1258.slice/crio-b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb WatchSource:0}: Error finding container b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb: Status 404 returned error can't find the container with id b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb
Oct 31 03:30:41.870046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:41.870021    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"]
Oct 31 03:30:42.464128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.464088    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"]
Oct 31 03:30:42.469574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.469542    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.476175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.476147    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"]
Oct 31 03:30:42.544533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.544493    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qxfw\" (UniqueName: \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.544734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.544639    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.544867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.544849    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.646337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.646297    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qxfw\" (UniqueName: \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.646535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.646452    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.646670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.646598    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.647262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.647231    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.649089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.649062    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.661504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.661473    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qxfw\" (UniqueName: \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\") pod \"maas-api-revoke-sub-404746c7-sljnk\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.772651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.772599    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh" event={"ID":"f4d79bea-3f93-462a-9255-460b4e0a1258","Type":"ContainerStarted","Data":"06e8b32ec6a663f474e9b6620eb77a0e1c96aaab9cbf47bf5cc2b9fc9419b552"}
Oct 31 03:30:42.772837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.772675    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh" event={"ID":"f4d79bea-3f93-462a-9255-460b4e0a1258","Type":"ContainerStarted","Data":"b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb"}
Oct 31 03:30:42.790934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.790872    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh" podStartSLOduration=1.790850194 podStartE2EDuration="1.790850194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:42.790694711 +0000 UTC m=+917.315914302" watchObservedRunningTime="2026-10-31 03:30:42.790850194 +0000 UTC m=+917.316069784"
Oct 31 03:30:42.798914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.798820    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:42.849916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.849875    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"]
Oct 31 03:30:42.873632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.867180    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:42.961288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:42.961107    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"]
Oct 31 03:30:43.017951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.017909    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.018239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.018202    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.018380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.018357    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29p48\" (UniqueName: \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.120468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.120364    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.120700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.120495    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.120700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.120574    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29p48\" (UniqueName: \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.121193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.121165    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.123111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.123076    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.142382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.142340    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29p48\" (UniqueName: \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\") pod \"maas-api-revoke-sub-e3e4ad13-szf9b\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.297887 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:43.297851    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d914a1_765f_45fb_8cbb_3ea94f495242.slice/crio-1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146 WatchSource:0}: Error finding container 1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146: Status 404 returned error can't find the container with id 1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146
Oct 31 03:30:43.299407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.299377    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"]
Oct 31 03:30:43.333983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.333937    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:43.636442 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:43.636413    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89e62ec8_6ee2_42dc_8c61_1e83cbe4eec1.slice/crio-1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856 WatchSource:0}: Error finding container 1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856: Status 404 returned error can't find the container with id 1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856
Oct 31 03:30:43.638014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.637982    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"]
Oct 31 03:30:43.813086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.813045    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b" event={"ID":"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1","Type":"ContainerStarted","Data":"1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856"}
Oct 31 03:30:43.815829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.815781    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk" event={"ID":"e2d914a1-765f-45fb-8cbb-3ea94f495242","Type":"ContainerStarted","Data":"c9fe9ffda4ae827dc3f94d42711bfa839b894e0cc98c3c374622562df4cf9831"}
Oct 31 03:30:43.816035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.815854    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk" event={"ID":"e2d914a1-765f-45fb-8cbb-3ea94f495242","Type":"ContainerStarted","Data":"1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146"}
Oct 31 03:30:43.818347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.818313    5098 generic.go:356] "Generic (PLEG): container finished" podID="f4d79bea-3f93-462a-9255-460b4e0a1258" containerID="06e8b32ec6a663f474e9b6620eb77a0e1c96aaab9cbf47bf5cc2b9fc9419b552" exitCode=0
Oct 31 03:30:43.818507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.818476    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh" event={"ID":"f4d79bea-3f93-462a-9255-460b4e0a1258","Type":"ContainerDied","Data":"06e8b32ec6a663f474e9b6620eb77a0e1c96aaab9cbf47bf5cc2b9fc9419b552"}
Oct 31 03:30:43.842038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:43.841969    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk" podStartSLOduration=1.841948265 podStartE2EDuration="1.841948265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:30:43.834625364 +0000 UTC m=+918.359844946" watchObservedRunningTime="2026-10-31 03:30:43.841948265 +0000 UTC m=+918.367167855"
Oct 31 03:30:44.830616 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:44.830568    5098 generic.go:356] "Generic (PLEG): container finished" podID="89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" containerID="f5fa9dab66725d9d73a8921bd1dd2a2907883d285e8d77ea2da0b7cc6cdb8a4f" exitCode=0
Oct 31 03:30:44.831018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:44.830638    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b" event={"ID":"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1","Type":"ContainerDied","Data":"f5fa9dab66725d9d73a8921bd1dd2a2907883d285e8d77ea2da0b7cc6cdb8a4f"}
Oct 31 03:30:44.832498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:44.832471    5098 generic.go:356] "Generic (PLEG): container finished" podID="e2d914a1-765f-45fb-8cbb-3ea94f495242" containerID="c9fe9ffda4ae827dc3f94d42711bfa839b894e0cc98c3c374622562df4cf9831" exitCode=0
Oct 31 03:30:44.832650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:44.832558    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk" event={"ID":"e2d914a1-765f-45fb-8cbb-3ea94f495242","Type":"ContainerDied","Data":"c9fe9ffda4ae827dc3f94d42711bfa839b894e0cc98c3c374622562df4cf9831"}
Oct 31 03:30:45.015836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.015754    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:45.053625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.053564    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\") pod \"f4d79bea-3f93-462a-9255-460b4e0a1258\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") "
Oct 31 03:30:45.053931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.053911    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\") pod \"f4d79bea-3f93-462a-9255-460b4e0a1258\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") "
Oct 31 03:30:45.054015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.053947    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\" (UniqueName: \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\") pod \"f4d79bea-3f93-462a-9255-460b4e0a1258\" (UID: \"f4d79bea-3f93-462a-9255-460b4e0a1258\") "
Oct 31 03:30:45.054080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.054018    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca" pod "f4d79bea-3f93-462a-9255-460b4e0a1258" (UID: "f4d79bea-3f93-462a-9255-460b4e0a1258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:45.054655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.054629    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:45.056406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.056377    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht" pod "f4d79bea-3f93-462a-9255-460b4e0a1258" (UID: "f4d79bea-3f93-462a-9255-460b4e0a1258"). InnerVolumeSpecName "kube-api-access-lddht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:45.056711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.056678    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca" pod "f4d79bea-3f93-462a-9255-460b4e0a1258" (UID: "f4d79bea-3f93-462a-9255-460b4e0a1258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:45.156488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.156450    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4d79bea-3f93-462a-9255-460b4e0a1258-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:45.156488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.156485    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lddht\" (UniqueName: \"kubernetes.io/projected/f4d79bea-3f93-462a-9255-460b4e0a1258-kube-api-access-lddht\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:45.856268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.856225    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh" event={"ID":"f4d79bea-3f93-462a-9255-460b4e0a1258","Type":"ContainerDied","Data":"b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb"}
Oct 31 03:30:45.856790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.856339    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ea14a884e929a2938f045d9725d47bdd6bc6904737cd0406f10f5b86144ceb"
Oct 31 03:30:45.857353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:45.856906    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"
Oct 31 03:30:46.151541 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.151514    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:46.154852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.154825    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:46.176204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176167    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\") pod \"e2d914a1-765f-45fb-8cbb-3ea94f495242\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") "
Oct 31 03:30:46.176394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176232    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\") pod \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") "
Oct 31 03:30:46.176466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176419    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\" (UniqueName: \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\") pod \"e2d914a1-765f-45fb-8cbb-3ea94f495242\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") "
Oct 31 03:30:46.176518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176472    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\") pod \"e2d914a1-765f-45fb-8cbb-3ea94f495242\" (UID: \"e2d914a1-765f-45fb-8cbb-3ea94f495242\") "
Oct 31 03:30:46.176596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176576    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\") pod \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") "
Oct 31 03:30:46.176679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176650    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca" pod "e2d914a1-765f-45fb-8cbb-3ea94f495242" (UID: "e2d914a1-765f-45fb-8cbb-3ea94f495242"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:46.176767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.176745    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\" (UniqueName: \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\") pod \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\" (UID: \"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1\") "
Oct 31 03:30:46.177517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.177470    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca" pod "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" (UID: "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:46.177633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.177546    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.179429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.179396    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw" pod "e2d914a1-765f-45fb-8cbb-3ea94f495242" (UID: "e2d914a1-765f-45fb-8cbb-3ea94f495242"). InnerVolumeSpecName "kube-api-access-6qxfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:46.179697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.179671    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca" pod "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" (UID: "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:46.179802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.179718    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48" pod "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" (UID: "89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1"). InnerVolumeSpecName "kube-api-access-29p48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:46.179846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.179828    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca" pod "e2d914a1-765f-45fb-8cbb-3ea94f495242" (UID: "e2d914a1-765f-45fb-8cbb-3ea94f495242"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:46.279360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.279326    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.279360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.279362    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qxfw\" (UniqueName: \"kubernetes.io/projected/e2d914a1-765f-45fb-8cbb-3ea94f495242-kube-api-access-6qxfw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.279561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.279375    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2d914a1-765f-45fb-8cbb-3ea94f495242-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.279561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.279385    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.279561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.279394    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29p48\" (UniqueName: \"kubernetes.io/projected/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1-kube-api-access-29p48\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:46.869528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.869496    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"
Oct 31 03:30:46.870025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.869531    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b" event={"ID":"89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1","Type":"ContainerDied","Data":"1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856"}
Oct 31 03:30:46.870025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.869585    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cb072d701ed7dcd43ed31a8e45da44f852a07a7df9e5f763e1afed402095856"
Oct 31 03:30:46.871267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.871237    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"
Oct 31 03:30:46.871404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.871265    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk" event={"ID":"e2d914a1-765f-45fb-8cbb-3ea94f495242","Type":"ContainerDied","Data":"1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146"}
Oct 31 03:30:46.871404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:46.871311    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1ec033ce244535643ccc07f6e3c9c1faea232c111592b4933575310358710146"
Oct 31 03:30:50.862877 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:50.862842    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:52.105385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.105328    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"]
Oct 31 03:30:52.106913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106880    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e2d914a1-765f-45fb-8cbb-3ea94f495242" containerName="revoke-keys"
Oct 31 03:30:52.106913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106911    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2d914a1-765f-45fb-8cbb-3ea94f495242" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106960    5098 cpu_manager.go:414] "containerMap: removing container" podUID="89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106969    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106987    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f4d79bea-3f93-462a-9255-460b4e0a1258" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.106994    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4d79bea-3f93-462a-9255-460b4e0a1258" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.107146    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2d914a1-765f-45fb-8cbb-3ea94f495242" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.107166    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" containerName="revoke-keys"
Oct 31 03:30:52.107530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.107175    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4d79bea-3f93-462a-9255-460b4e0a1258" containerName="revoke-keys"
Oct 31 03:30:52.113486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.113459    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"]
Oct 31 03:30:52.113643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.113579    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.118308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.118283    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:30:52.119894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.119872    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:30:52.121085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.120059    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:30:52.121085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.120438    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:30:52.121240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.121018    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.121299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.121253    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlv5m\" (UniqueName: \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.121401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.121375    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.222587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.222551    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.222776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.222714    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlv5m\" (UniqueName: \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.222818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.222788    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.223428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.223398    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.225207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.225174    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.231516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.231486    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlv5m\" (UniqueName: \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\") pod \"maas-api-revoke-sub-13fbda30-7kl4q\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.441689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.436029    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:52.792738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.792708    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"]
Oct 31 03:30:52.794676 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:52.794652    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e92cfe_71a7_4886_b8c3_bd328ccd9e76.slice/crio-3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977 WatchSource:0}: Error finding container 3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977: Status 404 returned error can't find the container with id 3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977
Oct 31 03:30:52.990860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:52.990821    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q" event={"ID":"37e92cfe-71a7-4886-b8c3-bd328ccd9e76","Type":"ContainerStarted","Data":"3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977"}
Oct 31 03:30:53.410194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.410100    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"]
Oct 31 03:30:53.415340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.415310    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.423313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.423279    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"]
Oct 31 03:30:53.546493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.546456    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdwx\" (UniqueName: \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.546493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.546497    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.546897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.546694    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.647895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.647858    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdwx\" (UniqueName: \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.648080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.647902    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.648080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.648048    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.648750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.648724    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.650723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.650696    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.659816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.659767    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdwx\" (UniqueName: \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\") pod \"maas-api-revoke-sub-5a9208d1-jslb8\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:53.735968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:53.735881    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:54.002005 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:54.001920    5098 generic.go:356] "Generic (PLEG): container finished" podID="37e92cfe-71a7-4886-b8c3-bd328ccd9e76" containerID="92075ce215d3c1b371b1efe85b844741b58cf12e1dbe41ceac1712fc6f63af48" exitCode=0
Oct 31 03:30:54.002167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:54.002004    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q" event={"ID":"37e92cfe-71a7-4886-b8c3-bd328ccd9e76","Type":"ContainerDied","Data":"92075ce215d3c1b371b1efe85b844741b58cf12e1dbe41ceac1712fc6f63af48"}
Oct 31 03:30:54.026163 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:54.026134    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd340b93f_df55_466c_bf1b_98da8d6cf842.slice/crio-6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491 WatchSource:0}: Error finding container 6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491: Status 404 returned error can't find the container with id 6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491
Oct 31 03:30:54.028481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:54.028459    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"]
Oct 31 03:30:55.013467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.013433    5098 generic.go:356] "Generic (PLEG): container finished" podID="d340b93f-df55-466c-bf1b-98da8d6cf842" containerID="ec57428fd99aa90c84b7ee1bccd11b55ff36d74e843954e71eb5d1f4ca170736" exitCode=0
Oct 31 03:30:55.013958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.013524    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8" event={"ID":"d340b93f-df55-466c-bf1b-98da8d6cf842","Type":"ContainerDied","Data":"ec57428fd99aa90c84b7ee1bccd11b55ff36d74e843954e71eb5d1f4ca170736"}
Oct 31 03:30:55.013958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.013582    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8" event={"ID":"d340b93f-df55-466c-bf1b-98da8d6cf842","Type":"ContainerStarted","Data":"6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491"}
Oct 31 03:30:55.138622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.137284    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"]
Oct 31 03:30:55.148070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.148034    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.160993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.160956    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"]
Oct 31 03:30:55.200212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.200184    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:55.281188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.281150    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwjw\" (UniqueName: \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.281369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.281316    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.281477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.281459    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.382516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.382481    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\") pod \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") "
Oct 31 03:30:55.382724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.382539    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\" (UniqueName: \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\") pod \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") "
Oct 31 03:30:55.382785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.382768    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\") pod \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\" (UID: \"37e92cfe-71a7-4886-b8c3-bd328ccd9e76\") "
Oct 31 03:30:55.382911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.382896    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.382986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.382925    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca" pod "37e92cfe-71a7-4886-b8c3-bd328ccd9e76" (UID: "37e92cfe-71a7-4886-b8c3-bd328ccd9e76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:55.383056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.383040    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwjw\" (UniqueName: \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.383267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.383247    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.383476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.383461    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:55.383941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.383922    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.384877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.384848    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m" pod "37e92cfe-71a7-4886-b8c3-bd328ccd9e76" (UID: "37e92cfe-71a7-4886-b8c3-bd328ccd9e76"). InnerVolumeSpecName "kube-api-access-tlv5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:55.385345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.385322    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca" pod "37e92cfe-71a7-4886-b8c3-bd328ccd9e76" (UID: "37e92cfe-71a7-4886-b8c3-bd328ccd9e76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:55.385547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.385530    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.391276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.391252    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwjw\" (UniqueName: \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\") pod \"maas-api-revoke-sub-946be3d5-r8tg5\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.484743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.484702    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:55.484743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.484732    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlv5m\" (UniqueName: \"kubernetes.io/projected/37e92cfe-71a7-4886-b8c3-bd328ccd9e76-kube-api-access-tlv5m\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:55.501498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.501470    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:55.797134 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:55.797099    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddeaa747_ce05_4b9a_860b_8915ded84ba4.slice/crio-aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828 WatchSource:0}: Error finding container aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828: Status 404 returned error can't find the container with id aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828
Oct 31 03:30:55.799591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:55.799562    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"]
Oct 31 03:30:56.025282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.025251    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"
Oct 31 03:30:56.025786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.025252    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q" event={"ID":"37e92cfe-71a7-4886-b8c3-bd328ccd9e76","Type":"ContainerDied","Data":"3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977"}
Oct 31 03:30:56.025786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.025381    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d6dd7e736cbfa9bbaa19236467c62befa1d3a19c869af857ef9a69dae6da977"
Oct 31 03:30:56.026741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.026714    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5" event={"ID":"ddeaa747-ce05-4b9a-860b-8915ded84ba4","Type":"ContainerStarted","Data":"aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828"}
Oct 31 03:30:56.178188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.178159    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:56.323031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.322991    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\") pod \"d340b93f-df55-466c-bf1b-98da8d6cf842\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") "
Oct 31 03:30:56.323218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.323198    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\") pod \"d340b93f-df55-466c-bf1b-98da8d6cf842\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") "
Oct 31 03:30:56.323301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.323257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\" (UniqueName: \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\") pod \"d340b93f-df55-466c-bf1b-98da8d6cf842\" (UID: \"d340b93f-df55-466c-bf1b-98da8d6cf842\") "
Oct 31 03:30:56.327157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.326941    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx" pod "d340b93f-df55-466c-bf1b-98da8d6cf842" (UID: "d340b93f-df55-466c-bf1b-98da8d6cf842"). InnerVolumeSpecName "kube-api-access-whdwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:56.328759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.327487    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca" pod "d340b93f-df55-466c-bf1b-98da8d6cf842" (UID: "d340b93f-df55-466c-bf1b-98da8d6cf842"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:56.338837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.338766    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca" pod "d340b93f-df55-466c-bf1b-98da8d6cf842" (UID: "d340b93f-df55-466c-bf1b-98da8d6cf842"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:56.346024 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:30:56.345977    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:30:56.352995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.352916    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"]
Oct 31 03:30:56.355381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355353    5098 cpu_manager.go:414] "containerMap: removing container" podUID="37e92cfe-71a7-4886-b8c3-bd328ccd9e76" containerName="revoke-keys"
Oct 31 03:30:56.355381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355381    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e92cfe-71a7-4886-b8c3-bd328ccd9e76" containerName="revoke-keys"
Oct 31 03:30:56.355548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355416    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d340b93f-df55-466c-bf1b-98da8d6cf842" containerName="revoke-keys"
Oct 31 03:30:56.355548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355424    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d340b93f-df55-466c-bf1b-98da8d6cf842" containerName="revoke-keys"
Oct 31 03:30:56.355838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355819    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e92cfe-71a7-4886-b8c3-bd328ccd9e76" containerName="revoke-keys"
Oct 31 03:30:56.355882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.355863    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d340b93f-df55-466c-bf1b-98da8d6cf842" containerName="revoke-keys"
Oct 31 03:30:56.368680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.368649    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.370241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.370216    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"]
Oct 31 03:30:56.426040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.426000    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:56.426040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.426035    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdwx\" (UniqueName: \"kubernetes.io/projected/d340b93f-df55-466c-bf1b-98da8d6cf842-kube-api-access-whdwx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:56.426251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.426053    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d340b93f-df55-466c-bf1b-98da8d6cf842-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:56.527044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.527011    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.527235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.527181    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.527348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.527333    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t7pv\" (UniqueName: \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.628918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.628823    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.629064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.628982    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t7pv\" (UniqueName: \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.629113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.629093    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.629650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.629599    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.631715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.631690    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.638056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.638031    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t7pv\" (UniqueName: \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\") pod \"maas-api-revoke-sub-a4ff7880-t2gdq\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:56.688570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:56.688526    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:57.044691 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:57.044645    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660c337e_6270_4021_a101_1345b352f5a9.slice/crio-7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c WatchSource:0}: Error finding container 7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c: Status 404 returned error can't find the container with id 7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c
Oct 31 03:30:57.047328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.046908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"]
Oct 31 03:30:57.047790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.047744    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8" event={"ID":"d340b93f-df55-466c-bf1b-98da8d6cf842","Type":"ContainerDied","Data":"6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491"}
Oct 31 03:30:57.047790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.047778    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"
Oct 31 03:30:57.047988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.047811    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6637459ae1aceb7394504e039b48f738687c1a65f22135d3d3f94a8859ac9491"
Oct 31 03:30:57.050274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.050249    5098 generic.go:356] "Generic (PLEG): container finished" podID="ddeaa747-ce05-4b9a-860b-8915ded84ba4" containerID="a5beb4710524c2479bd96201d603dedc916024747140156c1794ce00cf77c78e" exitCode=0
Oct 31 03:30:57.050392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:57.050378    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5" event={"ID":"ddeaa747-ce05-4b9a-860b-8915ded84ba4","Type":"ContainerDied","Data":"a5beb4710524c2479bd96201d603dedc916024747140156c1794ce00cf77c78e"}
Oct 31 03:30:58.061284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.061248    5098 generic.go:356] "Generic (PLEG): container finished" podID="660c337e-6270-4021-a101-1345b352f5a9" containerID="239bed7a869ff3d61e6726a5b78c25842211bea5890f5db347637bdd7ac8d3ee" exitCode=0
Oct 31 03:30:58.061750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.061333    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq" event={"ID":"660c337e-6270-4021-a101-1345b352f5a9","Type":"ContainerDied","Data":"239bed7a869ff3d61e6726a5b78c25842211bea5890f5db347637bdd7ac8d3ee"}
Oct 31 03:30:58.061750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.061393    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq" event={"ID":"660c337e-6270-4021-a101-1345b352f5a9","Type":"ContainerStarted","Data":"7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c"}
Oct 31 03:30:58.232085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.232012    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:58.250389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.250350    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\") pod \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") "
Oct 31 03:30:58.250565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.250431    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\") pod \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") "
Oct 31 03:30:58.250680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.250655    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\" (UniqueName: \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\") pod \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\" (UID: \"ddeaa747-ce05-4b9a-860b-8915ded84ba4\") "
Oct 31 03:30:58.255424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.255370    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca" pod "ddeaa747-ce05-4b9a-860b-8915ded84ba4" (UID: "ddeaa747-ce05-4b9a-860b-8915ded84ba4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:58.257011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.256978    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw" pod "ddeaa747-ce05-4b9a-860b-8915ded84ba4" (UID: "ddeaa747-ce05-4b9a-860b-8915ded84ba4"). InnerVolumeSpecName "kube-api-access-pvwjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:58.257675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.257643    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca" pod "ddeaa747-ce05-4b9a-860b-8915ded84ba4" (UID: "ddeaa747-ce05-4b9a-860b-8915ded84ba4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:58.369202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.366917    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:58.369202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.366960    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddeaa747-ce05-4b9a-860b-8915ded84ba4-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:58.369202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.366978    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwjw\" (UniqueName: \"kubernetes.io/projected/ddeaa747-ce05-4b9a-860b-8915ded84ba4-kube-api-access-pvwjw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:58.777893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.777857    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:30:58.866028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.865989    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"]
Oct 31 03:30:58.866826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.866806    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ddeaa747-ce05-4b9a-860b-8915ded84ba4" containerName="revoke-keys"
Oct 31 03:30:58.866826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.866826    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddeaa747-ce05-4b9a-860b-8915ded84ba4" containerName="revoke-keys"
Oct 31 03:30:58.866953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.866944    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddeaa747-ce05-4b9a-860b-8915ded84ba4" containerName="revoke-keys"
Oct 31 03:30:58.871781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.871750    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.873863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.873470    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.873863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.873675    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.873863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.873828    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgpgl\" (UniqueName: \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.881861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.881824    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"]
Oct 31 03:30:58.975731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.975682    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.975974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.975842    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.976044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.976008    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgpgl\" (UniqueName: \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.976644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.976579    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.978404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.978377    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:58.985138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:58.985110    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgpgl\" (UniqueName: \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\") pod \"maas-api-revoke-sub-32b4b2ea-gtf6p\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:59.102022 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.101934    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"
Oct 31 03:30:59.102513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.102140    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5" event={"ID":"ddeaa747-ce05-4b9a-860b-8915ded84ba4","Type":"ContainerDied","Data":"aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828"}
Oct 31 03:30:59.102513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.102201    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aecfafb315b4571a8637b9c91c56d08ec6a0bb9c1a1991dd90edce5efea2e828"
Oct 31 03:30:59.206740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.206702    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:30:59.266779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.266746    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:30:59.282728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.280571    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\") pod \"660c337e-6270-4021-a101-1345b352f5a9\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") "
Oct 31 03:30:59.282728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.280809    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\") pod \"660c337e-6270-4021-a101-1345b352f5a9\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") "
Oct 31 03:30:59.282728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.280865    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\" (UniqueName: \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\") pod \"660c337e-6270-4021-a101-1345b352f5a9\" (UID: \"660c337e-6270-4021-a101-1345b352f5a9\") "
Oct 31 03:30:59.282728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.282353    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca" pod "660c337e-6270-4021-a101-1345b352f5a9" (UID: "660c337e-6270-4021-a101-1345b352f5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:30:59.306231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.306151    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv" pod "660c337e-6270-4021-a101-1345b352f5a9" (UID: "660c337e-6270-4021-a101-1345b352f5a9"). InnerVolumeSpecName "kube-api-access-2t7pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:30:59.308765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.308731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca" pod "660c337e-6270-4021-a101-1345b352f5a9" (UID: "660c337e-6270-4021-a101-1345b352f5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:30:59.383335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.383233    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660c337e-6270-4021-a101-1345b352f5a9-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:59.383335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.383273    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660c337e-6270-4021-a101-1345b352f5a9-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:59.383335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.383294    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t7pv\" (UniqueName: \"kubernetes.io/projected/660c337e-6270-4021-a101-1345b352f5a9-kube-api-access-2t7pv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:30:59.518618 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:30:59.518562    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod585adb43_7243_4f67_aa58_5b37ef216ff9.slice/crio-a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a WatchSource:0}: Error finding container a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a: Status 404 returned error can't find the container with id a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a
Oct 31 03:30:59.520032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:30:59.520003    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"]
Oct 31 03:31:00.185571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.184630    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq" event={"ID":"660c337e-6270-4021-a101-1345b352f5a9","Type":"ContainerDied","Data":"7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c"}
Oct 31 03:31:00.185571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.184704    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2014db673b4180719d50dbdf794194561a1f8fe4b8018c70146a307312f57c"
Oct 31 03:31:00.185571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.184793    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"
Oct 31 03:31:00.222723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.221730    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p" event={"ID":"585adb43-7243-4f67-aa58-5b37ef216ff9","Type":"ContainerStarted","Data":"a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a"}
Oct 31 03:31:00.884766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.884704    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:00.942903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:00.930150    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:00.982632 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:00.979341    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:01.216900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.215667    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.334865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.320449    5098 generic.go:356] "Generic (PLEG): container finished" podID="585adb43-7243-4f67-aa58-5b37ef216ff9" containerID="873229428f5899ff8a81d059d99efe3471c93b01fed57a67db2a5e9762d9188d" exitCode=0
Oct 31 03:31:01.334865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.320529    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p" event={"ID":"585adb43-7243-4f67-aa58-5b37ef216ff9","Type":"ContainerDied","Data":"873229428f5899ff8a81d059d99efe3471c93b01fed57a67db2a5e9762d9188d"}
Oct 31 03:31:01.367547 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:01.367502    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:31:01.462539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.461740    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.488144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.484706    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.618633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.602697    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:01.701896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.700681    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.901931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.891914    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.901931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.892714    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:31:01.937635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.936905    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:01.947296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:01.938405    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:31:02.262771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.252541    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:02.262771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.253116    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:31:02.314631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.302222    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:02.707739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.706081    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.721275    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.729589    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.808999    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.815919    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.816326    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.894369    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:31:02.991959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:02.972225    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:03.029634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:03.015674    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:31:03.412891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:03.412842    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:03.536729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:03.519366    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:31:03.754327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:03.754206    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:04.061724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.052380    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:04.126337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.059761    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:04.126337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.105633    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:04.126337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.105858    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:04.206084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.190739    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:04.303493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.296268    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:31:04.306185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.306138    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:04.437821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.437769    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:31:04.746270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746196    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:31:04.746270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746258    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:31:04.746433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746295    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:31:04.746433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746374    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:31:04.746433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746408    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:31:04.746528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.746435    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"]
Oct 31 03:31:04.747493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.747467    5098 cpu_manager.go:414] "containerMap: removing container" podUID="660c337e-6270-4021-a101-1345b352f5a9" containerName="revoke-keys"
Oct 31 03:31:04.747493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.747490    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660c337e-6270-4021-a101-1345b352f5a9" containerName="revoke-keys"
Oct 31 03:31:04.747733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.747685    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="660c337e-6270-4021-a101-1345b352f5a9" containerName="revoke-keys"
Oct 31 03:31:04.763713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.752745    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:31:04.763713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.757740    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.782935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.782906    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"]
Oct 31 03:31:04.783171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.783153    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:31:04.783304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.783292    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:31:04.783454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.783439    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:31:04.784501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.784480    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.817069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.817015    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.817332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.817312    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.817537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.817514    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.818874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.818825    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.819140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.819123    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wftfk\" (UniqueName: \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.819305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.819282    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c26nm\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.819655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.819633    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.819848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.819832    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.819967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.819954    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.820155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.820143    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.820419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.820401    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.820534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.820521    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.887429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.887383    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:31:04.920081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.918810    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:04.921653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921573    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.921653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921627    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.921845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921686    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.921845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921772    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.921845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921820    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wftfk\" (UniqueName: \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.922058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921848    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c26nm\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.922058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921914    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.922058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.921970    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.922058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.922006    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.922375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.922353    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.922492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.922478    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.922549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.922516    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.923167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.923135    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.924391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.924360    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.925104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.925046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.925974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.925945    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.926361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.926333    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.926652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.926593    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.926920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.926892    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.928135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.928112    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.928943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.928912    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:31:04.929348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.929328    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.932014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.931936    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:31:04.952655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.952230    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wftfk\" (UniqueName: \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\") pod \"maas-api-revoke-sub-857b876a-dwfwl\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:04.956592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.952834    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:31:04.956592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.953991    5098 cpu_manager.go:414] "containerMap: removing container" podUID="585adb43-7243-4f67-aa58-5b37ef216ff9" containerName="revoke-keys"
Oct 31 03:31:04.956592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.954010    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="585adb43-7243-4f67-aa58-5b37ef216ff9" containerName="revoke-keys"
Oct 31 03:31:04.956592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.954163    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="585adb43-7243-4f67-aa58-5b37ef216ff9" containerName="revoke-keys"
Oct 31 03:31:04.958052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.957462    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c26nm\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.958052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.957969    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:04.960062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.960037    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:04.977949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:04.977920    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.026211    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\" (UniqueName: \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\") pod \"585adb43-7243-4f67-aa58-5b37ef216ff9\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") "
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.026345    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\") pod \"585adb43-7243-4f67-aa58-5b37ef216ff9\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") "
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.026514    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\") pod \"585adb43-7243-4f67-aa58-5b37ef216ff9\" (UID: \"585adb43-7243-4f67-aa58-5b37ef216ff9\") "
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.026697    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca" pod "585adb43-7243-4f67-aa58-5b37ef216ff9" (UID: "585adb43-7243-4f67-aa58-5b37ef216ff9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027057    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027104    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027266    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027318    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027361    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027408    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9pnd\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027450    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027545    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027631    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027690    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027718    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027774    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cr9tr\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027860    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027915    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.027953    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.028013    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.028039    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.028084    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.028225    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:05.032823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.031781    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca" pod "585adb43-7243-4f67-aa58-5b37ef216ff9" (UID: "585adb43-7243-4f67-aa58-5b37ef216ff9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:05.034047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.033659    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl" pod "585adb43-7243-4f67-aa58-5b37ef216ff9" (UID: "585adb43-7243-4f67-aa58-5b37ef216ff9"). InnerVolumeSpecName "kube-api-access-zgpgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:05.097013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.096971    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:05.109883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.109791    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:05.140775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.140503    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.140775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.140554    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.140775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.140680    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.141327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.140733    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.141327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141160    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.141499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141417    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.141564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141500    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.141637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141558    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9pnd\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.141637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141624    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.141736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141707    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.141798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.141779    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.144870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.144845    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.145068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145046    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.145210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145187    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.145289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.144384    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.145289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.142452    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.145434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145411    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cr9tr\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.145512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.143827    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.145589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.143166    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.145666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145486    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.145843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145823    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.145997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.145976    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.146120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.146103    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.146291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.146274    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.146686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.146666    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.146846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.146828    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.147073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.147045    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.147174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.147064    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgpgl\" (UniqueName: \"kubernetes.io/projected/585adb43-7243-4f67-aa58-5b37ef216ff9-kube-api-access-zgpgl\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:05.147278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.147231    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.147330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.147255    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/585adb43-7243-4f67-aa58-5b37ef216ff9-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:05.147422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.147395    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.150058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.150038    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.153012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.150547    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.153012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.151644    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.153012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.152050    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9pnd\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.154071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.154041    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.154458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.154412    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cr9tr\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.161816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.161785    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\") pod \"e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") " pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.306704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.306666    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:05.316433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.316398    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:05.408170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.408108    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p" event={"ID":"585adb43-7243-4f67-aa58-5b37ef216ff9","Type":"ContainerDied","Data":"a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a"}
Oct 31 03:31:05.408339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.408221    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1d255c2d2335aa990c0011d171615acafcccf2ebaaaf5c7efbf88944b68083a"
Oct 31 03:31:05.408339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.408312    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"
Oct 31 03:31:05.468737 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:05.468700    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd343ef19_7452_46cc_9c9c_5241004fa965.slice/crio-d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94 WatchSource:0}: Error finding container d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94: Status 404 returned error can't find the container with id d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94
Oct 31 03:31:05.469923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.469893    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"]
Oct 31 03:31:05.481321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.481290    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:31:05.505304 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:05.505273    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeafcb788_673c_4aaa_8b58_2e6defe730b1.slice/crio-9c31c466e0eeed54118200ba35e48a327d8a3da75053000acadd86e418ef2295 WatchSource:0}: Error finding container 9c31c466e0eeed54118200ba35e48a327d8a3da75053000acadd86e418ef2295: Status 404 returned error can't find the container with id 9c31c466e0eeed54118200ba35e48a327d8a3da75053000acadd86e418ef2295
Oct 31 03:31:05.513159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.513121    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.513286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.513209    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.513286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.513256    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.732119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.732089    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"]
Oct 31 03:31:05.739401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.738194    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.746561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.746512    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"]
Oct 31 03:31:05.772028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.769016    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:31:05.775655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.775530    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:31:05.776079 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:05.776053    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1d2074_b249_4424_8c9e_0f4ca5551a68.slice/crio-f109a854514b592bd2a61a8a7c85143a7da8c0ee71422bac789af330497e253e WatchSource:0}: Error finding container f109a854514b592bd2a61a8a7c85143a7da8c0ee71422bac789af330497e253e: Status 404 returned error can't find the container with id f109a854514b592bd2a61a8a7c85143a7da8c0ee71422bac789af330497e253e
Oct 31 03:31:05.778478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.777156    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.778478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.778334    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.778478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.778384    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.779347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.779316    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.779560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.779532    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.779675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.779589    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:05.894322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.893945    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xzt\" (UniqueName: \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.894322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.894035    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.894322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.894061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.996644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.995702    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xzt\" (UniqueName: \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.996644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.995799    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.996644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.995831    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:05.996939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:05.996900    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:06.000434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.000400    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:06.012655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.009806    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xzt\" (UniqueName: \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\") pod \"maas-api-revoke-sub-4caa91bb-8fmtc\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:06.084630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.078882    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:06.413339 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:06.413307    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b2a915_1c4c_4d7e_8d46_3bb134a46029.slice/crio-4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024 WatchSource:0}: Error finding container 4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024: Status 404 returned error can't find the container with id 4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024
Oct 31 03:31:06.414763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.414640    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"]
Oct 31 03:31:06.425368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.425316    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" event={"ID":"eafcb788-673c-4aaa-8b58-2e6defe730b1","Type":"ContainerStarted","Data":"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"}
Oct 31 03:31:06.425543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.425416    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" event={"ID":"eafcb788-673c-4aaa-8b58-2e6defe730b1","Type":"ContainerStarted","Data":"9c31c466e0eeed54118200ba35e48a327d8a3da75053000acadd86e418ef2295"}
Oct 31 03:31:06.430669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.428142    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" event={"ID":"e74b0119-7795-4e12-9a3f-9e3824fa0df1","Type":"ContainerStarted","Data":"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"}
Oct 31 03:31:06.430669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.428219    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" event={"ID":"e74b0119-7795-4e12-9a3f-9e3824fa0df1","Type":"ContainerStarted","Data":"5fb3f1e208fa3369b21ac23a8b46d95e886f9ef4d00ff77b5a6a961db837fee1"}
Oct 31 03:31:06.433128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.433095    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl" event={"ID":"d343ef19-7452-46cc-9c9c-5241004fa965","Type":"ContainerStarted","Data":"b86432d64c21a4b345b54db79267bd7e77b3c0793e942112eec183180e2b39cb"}
Oct 31 03:31:06.433243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.433153    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl" event={"ID":"d343ef19-7452-46cc-9c9c-5241004fa965","Type":"ContainerStarted","Data":"d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94"}
Oct 31 03:31:06.436474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.436446    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" event={"ID":"6e1d2074-b249-4424-8c9e-0f4ca5551a68","Type":"ContainerStarted","Data":"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"}
Oct 31 03:31:06.436579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.436499    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" event={"ID":"6e1d2074-b249-4424-8c9e-0f4ca5551a68","Type":"ContainerStarted","Data":"f109a854514b592bd2a61a8a7c85143a7da8c0ee71422bac789af330497e253e"}
Oct 31 03:31:06.466818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.466758    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podStartSLOduration=2.466736231 podStartE2EDuration="2.466736231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.446709777 +0000 UTC m=+940.971929360" watchObservedRunningTime="2026-10-31 03:31:06.466736231 +0000 UTC m=+940.991955821"
Oct 31 03:31:06.507871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.507816    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podStartSLOduration=2.507788156 podStartE2EDuration="2.507788156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.48038964 +0000 UTC m=+941.005609221" watchObservedRunningTime="2026-10-31 03:31:06.507788156 +0000 UTC m=+941.033007819"
Oct 31 03:31:06.521008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.520950    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl" podStartSLOduration=2.520934112 podStartE2EDuration="2.520934112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.506714616 +0000 UTC m=+941.031934206" watchObservedRunningTime="2026-10-31 03:31:06.520934112 +0000 UTC m=+941.046153700"
Oct 31 03:31:06.551675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:06.546160    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podStartSLOduration=2.5461297800000002 podStartE2EDuration="2.54612978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:06.532759735 +0000 UTC m=+941.057979335" watchObservedRunningTime="2026-10-31 03:31:06.54612978 +0000 UTC m=+941.071349370"
Oct 31 03:31:07.132642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.130706    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:07.133685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.133542    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:07.133685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.133601    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:07.310718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.310671    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:07.315636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.315525    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:07.315636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.315597    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:07.327248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.327145    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:07.327596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.327557    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:07.327834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.327803    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:07.529405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.529344    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc" event={"ID":"34b2a915-1c4c-4d7e-8d46-3bb134a46029","Type":"ContainerStarted","Data":"a6a26bfe33077cdd5a7942c2efdbb3034cc96d5118e90b27ad1664944ee2a0eb"}
Oct 31 03:31:07.533442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.533408    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc" event={"ID":"34b2a915-1c4c-4d7e-8d46-3bb134a46029","Type":"ContainerStarted","Data":"4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024"}
Oct 31 03:31:07.548072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.543721    5098 generic.go:356] "Generic (PLEG): container finished" podID="d343ef19-7452-46cc-9c9c-5241004fa965" containerID="b86432d64c21a4b345b54db79267bd7e77b3c0793e942112eec183180e2b39cb" exitCode=0
Oct 31 03:31:07.548072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.543850    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl" event={"ID":"d343ef19-7452-46cc-9c9c-5241004fa965","Type":"ContainerDied","Data":"b86432d64c21a4b345b54db79267bd7e77b3c0793e942112eec183180e2b39cb"}
Oct 31 03:31:07.663704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:07.659180    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc" podStartSLOduration=2.659156818 podStartE2EDuration="2.659156818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:07.573924329 +0000 UTC m=+942.099143920" watchObservedRunningTime="2026-10-31 03:31:07.659156818 +0000 UTC m=+942.184376415"
Oct 31 03:31:08.113631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.112682    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:08.113631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.112773    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:08.308134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.308093    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:08.308883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.308433    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:08.323826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.323781    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:08.324055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.323980    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:08.628631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.625310    5098 generic.go:356] "Generic (PLEG): container finished" podID="34b2a915-1c4c-4d7e-8d46-3bb134a46029" containerID="a6a26bfe33077cdd5a7942c2efdbb3034cc96d5118e90b27ad1664944ee2a0eb" exitCode=0
Oct 31 03:31:08.628631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:08.627024    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc" event={"ID":"34b2a915-1c4c-4d7e-8d46-3bb134a46029","Type":"ContainerDied","Data":"a6a26bfe33077cdd5a7942c2efdbb3034cc96d5118e90b27ad1664944ee2a0eb"}
Oct 31 03:31:09.145673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.145260    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:09.145673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.145327    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:09.351632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.342895    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:09.351632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.342964    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:09.351632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.343383    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:09.351632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.343425    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:09.385474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.385433    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:09.554711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.549493    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\") pod \"d343ef19-7452-46cc-9c9c-5241004fa965\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") "
Oct 31 03:31:09.554711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.549565    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\") pod \"d343ef19-7452-46cc-9c9c-5241004fa965\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") "
Oct 31 03:31:09.695325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.606876    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca" pod "d343ef19-7452-46cc-9c9c-5241004fa965" (UID: "d343ef19-7452-46cc-9c9c-5241004fa965"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:09.786145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.769770    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\" (UniqueName: \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\") pod \"d343ef19-7452-46cc-9c9c-5241004fa965\" (UID: \"d343ef19-7452-46cc-9c9c-5241004fa965\") "
Oct 31 03:31:09.786145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.770855    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:09.841694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.839900    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7264-46ccf5"]
Oct 31 03:31:09.841694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.840684    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca" pod "d343ef19-7452-46cc-9c9c-5241004fa965" (UID: "d343ef19-7452-46cc-9c9c-5241004fa965"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:09.841694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.841101    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d343ef19-7452-46cc-9c9c-5241004fa965" containerName="revoke-keys"
Oct 31 03:31:09.841694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.841126    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d343ef19-7452-46cc-9c9c-5241004fa965" containerName="revoke-keys"
Oct 31 03:31:09.841694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.841376    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d343ef19-7452-46cc-9c9c-5241004fa965" containerName="revoke-keys"
Oct 31 03:31:09.843869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.843844    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"
Oct 31 03:31:09.846843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.846814    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk" pod "d343ef19-7452-46cc-9c9c-5241004fa965" (UID: "d343ef19-7452-46cc-9c9c-5241004fa965"). InnerVolumeSpecName "kube-api-access-wftfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:09.879282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.877736    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d343ef19-7452-46cc-9c9c-5241004fa965-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:09.879282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.877775    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wftfk\" (UniqueName: \"kubernetes.io/projected/d343ef19-7452-46cc-9c9c-5241004fa965-kube-api-access-wftfk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:09.879724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.879676    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl" event={"ID":"d343ef19-7452-46cc-9c9c-5241004fa965","Type":"ContainerDied","Data":"d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94"}
Oct 31 03:31:09.879922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.879905    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d73d070f58e8cfc4772c625286d1f5bfc83381f8a9d1cd5b38c80f82b0e41c94"
Oct 31 03:31:09.880116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.880103    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:09.943320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.941706    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xd6kv\""
Oct 31 03:31:09.948192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.948154    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7264-46ccf5"]
Oct 31 03:31:10.003147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:09.986688    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msvnb\" (UniqueName: \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\") pod \"test-curl-7264-46ccf5\" (UID: \"9640791f-1a44-45f0-8931-218c90f0a671\") " pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:10.140793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.140688    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msvnb\" (UniqueName: \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\") pod \"test-curl-7264-46ccf5\" (UID: \"9640791f-1a44-45f0-8931-218c90f0a671\") " pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:10.142908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.142856    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:10.145143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.145087    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:10.345657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.322032    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:10.345657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.322109    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:10.491091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.491050    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:10.491414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.491367    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:10.542667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.537779    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msvnb\" (UniqueName: \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\") pod \"test-curl-7264-46ccf5\" (UID: \"9640791f-1a44-45f0-8931-218c90f0a671\") " pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:10.542667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:10.538115    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:11.044672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.044640    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"]
Oct 31 03:31:11.133928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.133840    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:11.134139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.133924    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:11.140076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.140039    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"]
Oct 31 03:31:11.140232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.140181    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.162663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.156836    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:11.244416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.244383    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.244833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.244814    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd5jl\" (UniqueName: \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.244951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.244941    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.348156    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.348244    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.361171    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.361238    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.362334    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\" (UniqueName: \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\") pod \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") "
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.380073    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\") pod \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") "
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.380209    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\") pod \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\" (UID: \"34b2a915-1c4c-4d7e-8d46-3bb134a46029\") "
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.380595    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.380959    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd5jl\" (UniqueName: \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.381015    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.410700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.383041    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.447905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.447855    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca" pod "34b2a915-1c4c-4d7e-8d46-3bb134a46029" (UID: "34b2a915-1c4c-4d7e-8d46-3bb134a46029"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:11.736782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.708108    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.747226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.747168    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt" pod "34b2a915-1c4c-4d7e-8d46-3bb134a46029" (UID: "34b2a915-1c4c-4d7e-8d46-3bb134a46029"). InnerVolumeSpecName "kube-api-access-b4xzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:11.844973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.824848    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca" pod "34b2a915-1c4c-4d7e-8d46-3bb134a46029" (UID: "34b2a915-1c4c-4d7e-8d46-3bb134a46029"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:11.844973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.844247    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b2a915-1c4c-4d7e-8d46-3bb134a46029-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.844973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.844271    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xzt\" (UniqueName: \"kubernetes.io/projected/34b2a915-1c4c-4d7e-8d46-3bb134a46029-kube-api-access-b4xzt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:11.899646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.848576    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.929635 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:11.926467    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:11.943873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:11.942978    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd5jl\" (UniqueName: \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\") pod \"maas-api-revoke-sub-17dae9d8-xbpcl\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:11.964402 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:11.964354    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:12.133644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.128366    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:12.135226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.135163    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:12.135388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.135294    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:12.142855 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.142822    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc" event={"ID":"34b2a915-1c4c-4d7e-8d46-3bb134a46029","Type":"ContainerDied","Data":"4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024"}
Oct 31 03:31:12.142981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.142884    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d22333e524bf6bee512566e0e4a4cea39536819660a71c1a96f162a13b2b024"
Oct 31 03:31:12.142981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.142952    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"
Oct 31 03:31:12.394377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.378313    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:12.394377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.378392    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:12.398507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.398388    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:12.398507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.398444    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:12.483646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.480818    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:31:12.665320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.665257    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7264-46ccf5"]
Oct 31 03:31:12.867797 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:12.867748    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2531a346_e183_4606_bfa1_c4179de689ee.slice/crio-4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038 WatchSource:0}: Error finding container 4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038: Status 404 returned error can't find the container with id 4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038
Oct 31 03:31:12.871150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:12.871119    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"]
Oct 31 03:31:13.125650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.118125    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused" start-of-body=
Oct 31 03:31:13.125650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.118190    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": dial tcp 10.132.0.135:15021: connect: connection refused"
Oct 31 03:31:13.209962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.209836    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7264-46ccf5" event={"ID":"9640791f-1a44-45f0-8931-218c90f0a671","Type":"ContainerStarted","Data":"04eac4da42dfd74c90c34640bc98abbeef7d4f401d01e32a4db214b1070a35e6"}
Oct 31 03:31:13.213431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.213377    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl" event={"ID":"2531a346-e183-4606-bfa1-c4179de689ee","Type":"ContainerStarted","Data":"4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038"}
Oct 31 03:31:13.307749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.307716    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused" start-of-body=
Oct 31 03:31:13.307989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.307961    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": dial tcp 10.132.0.136:15021: connect: connection refused"
Oct 31 03:31:13.317371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.317338    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body=
Oct 31 03:31:13.317549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:13.317398    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused"
Oct 31 03:31:13.572874 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:13.572825    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:13.575022 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:13.574983    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:14.231779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.231129    5098 generic.go:356] "Generic (PLEG): container finished" podID="2531a346-e183-4606-bfa1-c4179de689ee" containerID="f9540ca1e9635d9e3e25797ccf1d1b3fff0de9c5c3e69d3573a97883e1bf3b0c" exitCode=0
Oct 31 03:31:14.231779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.231304    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl" event={"ID":"2531a346-e183-4606-bfa1-c4179de689ee","Type":"ContainerDied","Data":"f9540ca1e9635d9e3e25797ccf1d1b3fff0de9c5c3e69d3573a97883e1bf3b0c"}
Oct 31 03:31:14.567019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.566955    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:14.568815 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.568785    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:14.570794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.570772    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:31:14.827723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.827371    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:14.828243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.828198    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:14.829590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.829559    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:31:14.844826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.844794    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:14.849453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.846105    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:14.849453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:14.848216    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:31:15.442726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.442692    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:15.628867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.628824    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\") pod \"2531a346-e183-4606-bfa1-c4179de689ee\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") "
Oct 31 03:31:15.635382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.635344    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\" (UniqueName: \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\") pod \"2531a346-e183-4606-bfa1-c4179de689ee\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") "
Oct 31 03:31:15.644080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.644044    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\") pod \"2531a346-e183-4606-bfa1-c4179de689ee\" (UID: \"2531a346-e183-4606-bfa1-c4179de689ee\") "
Oct 31 03:31:15.645552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.630935    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca" pod "2531a346-e183-4606-bfa1-c4179de689ee" (UID: "2531a346-e183-4606-bfa1-c4179de689ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:15.645552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.645507    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:15.669247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.667153    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca" pod "2531a346-e183-4606-bfa1-c4179de689ee" (UID: "2531a346-e183-4606-bfa1-c4179de689ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:15.669247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.668132    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl" pod "2531a346-e183-4606-bfa1-c4179de689ee" (UID: "2531a346-e183-4606-bfa1-c4179de689ee"). InnerVolumeSpecName "kube-api-access-nd5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:15.747412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.747373    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd5jl\" (UniqueName: \"kubernetes.io/projected/2531a346-e183-4606-bfa1-c4179de689ee-kube-api-access-nd5jl\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:15.747412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:15.747412    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2531a346-e183-4606-bfa1-c4179de689ee-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:16.298694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.298649    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl" event={"ID":"2531a346-e183-4606-bfa1-c4179de689ee","Type":"ContainerDied","Data":"4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038"}
Oct 31 03:31:16.298875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.298697    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"
Oct 31 03:31:16.298875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.298742    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a3f94abfac9f95dd0260f2da896df8ee502b68df5f7a37e6ee3fba829d56038"
Oct 31 03:31:16.968458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.968364    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"]
Oct 31 03:31:16.969122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969104    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2531a346-e183-4606-bfa1-c4179de689ee" containerName="revoke-keys"
Oct 31 03:31:16.969170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969122    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2531a346-e183-4606-bfa1-c4179de689ee" containerName="revoke-keys"
Oct 31 03:31:16.969170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969164    5098 cpu_manager.go:414] "containerMap: removing container" podUID="34b2a915-1c4c-4d7e-8d46-3bb134a46029" containerName="revoke-keys"
Oct 31 03:31:16.969170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969169    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b2a915-1c4c-4d7e-8d46-3bb134a46029" containerName="revoke-keys"
Oct 31 03:31:16.969309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969295    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2531a346-e183-4606-bfa1-c4179de689ee" containerName="revoke-keys"
Oct 31 03:31:16.969355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.969322    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b2a915-1c4c-4d7e-8d46-3bb134a46029" containerName="revoke-keys"
Oct 31 03:31:16.973478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.973458    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:16.981436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.981348    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:16.982988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.982962    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:16.983134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.983056    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:31:16.983204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.983133    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:16.983692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:16.983670    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"]
Oct 31 03:31:17.080573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.080539    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.080860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.080834    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49f4q\" (UniqueName: \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.080931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.080874    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.214628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.204125    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49f4q\" (UniqueName: \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.214628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.204211    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.227679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.219550    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.227679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.220623    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.295151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.263205    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.295151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.268240    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49f4q\" (UniqueName: \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\") pod \"maas-api-revoke-sub-c3444c45-86bvd\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.295151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.292982    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:17.321398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.321359    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7264-46ccf5" event={"ID":"9640791f-1a44-45f0-8931-218c90f0a671","Type":"ContainerStarted","Data":"7077d226c9852414b1386e9471b921cf5aa556a542dc375bd63b71fc5f068a29"}
Oct 31 03:31:17.347713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.347599    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7264-46ccf5" podStartSLOduration=4.357516891 podStartE2EDuration="8.347584005s" totalImagesPullingTime="3.990067114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:17.345283788 +0000 UTC m=+951.870503396" watchObservedRunningTime="2026-10-31 03:31:17.347584005 +0000 UTC m=+951.872803633"
Oct 31 03:31:17.615921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:17.615893    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"]
Oct 31 03:31:18.073653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.073592    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7264-46ccf5"]
Oct 31 03:31:18.080910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.080879    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7264-46ccf5"]
Oct 31 03:31:18.246506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.246425    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7264-189b9e"]
Oct 31 03:31:18.247419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.247394    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9640791f-1a44-45f0-8931-218c90f0a671" containerName="test-curl-7264-46ccf5"
Oct 31 03:31:18.247419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.247418    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9640791f-1a44-45f0-8931-218c90f0a671" containerName="test-curl-7264-46ccf5"
Oct 31 03:31:18.247577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.247567    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9640791f-1a44-45f0-8931-218c90f0a671" containerName="test-curl-7264-46ccf5"
Oct 31 03:31:18.258684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.252295    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:18.259742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.259715    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7264-189b9e"]
Oct 31 03:31:18.338906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.338866    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd" event={"ID":"16e5fdec-290d-4826-9757-2282b078121c","Type":"ContainerStarted","Data":"9165142a80acff69010639cd300c8e6b5b95d2f650f826a4fb3f0f037511e7c4"}
Oct 31 03:31:18.339125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.338928    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd" event={"ID":"16e5fdec-290d-4826-9757-2282b078121c","Type":"ContainerStarted","Data":"f34776f7e9dc678963d7e97dc9904fc7f314514693d4cc25ed6315c1e8a48500"}
Oct 31 03:31:18.344733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.344690    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cbjd\" (UniqueName: \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\") pod \"test-curl-7264-189b9e\" (UID: \"9fb035da-b9b8-4a80-91c3-2eaca6672d83\") " pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:18.369508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.369437    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd" podStartSLOduration=2.3694156250000002 podStartE2EDuration="2.369415625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:18.366049553 +0000 UTC m=+952.891269141" watchObservedRunningTime="2026-10-31 03:31:18.369415625 +0000 UTC m=+952.894635217"
Oct 31 03:31:18.446749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.446697    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cbjd\" (UniqueName: \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\") pod \"test-curl-7264-189b9e\" (UID: \"9fb035da-b9b8-4a80-91c3-2eaca6672d83\") " pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:18.469665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.469624    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:31:18.473673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.473593    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cbjd\" (UniqueName: \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\") pod \"test-curl-7264-189b9e\" (UID: \"9fb035da-b9b8-4a80-91c3-2eaca6672d83\") " pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:18.491952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.481668    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.491952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.491134    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-ed9442ec-gw-openshift-default-dockercfg-w8ldd\""
Oct 31 03:31:18.491952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.491449    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:31:18.569786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.569743    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.650671    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.650764    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpbtw\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.650900    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651059    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651097    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651214    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651253    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651287    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.656250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.651338    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.775658    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.775834    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.775859    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.775998    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.776036    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.776062    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.776106    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.776163    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.776230    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpbtw\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.778757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.778474    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.779527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.778820    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.779527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.779066    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.779527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.779498    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.793644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.779769    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.826705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.818990    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.826705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.820199    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.853633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.850174    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:18.853633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:18.851318    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpbtw\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\") pod \"e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") " pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:19.196554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:19.196464    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:19.752577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:19.751702    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.169386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.167810    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:31:20.178031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.176110    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.194119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.194051    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.197291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.195857    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.333656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.317104    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7264-189b9e"]
Oct 31 03:31:20.776727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.755845    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.776727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.756359    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:20.776727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:20.769440    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:31:21.175640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.155646    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:21.245119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.178104    5098 generic.go:356] "Generic (PLEG): container finished" podID="16e5fdec-290d-4826-9757-2282b078121c" containerID="9165142a80acff69010639cd300c8e6b5b95d2f650f826a4fb3f0f037511e7c4" exitCode=0
Oct 31 03:31:21.273629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.178153    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd" event={"ID":"16e5fdec-290d-4826-9757-2282b078121c","Type":"ContainerDied","Data":"9165142a80acff69010639cd300c8e6b5b95d2f650f826a4fb3f0f037511e7c4"}
Oct 31 03:31:21.274017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.178820    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:21.274577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.178878    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:21.288646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.275418    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:31:21.289061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.289032    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:31:21.697717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.200086    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:21.699839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.200125    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:21.812929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.812842    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7264-46ccf5" podUID="9640791f-1a44-45f0-8931-218c90f0a671" containerName="test-curl-7264-46ccf5" containerID="cri-o://7077d226c9852414b1386e9471b921cf5aa556a542dc375bd63b71fc5f068a29" gracePeriod=2
Oct 31 03:31:21.944632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.924981    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:21.964690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.960009    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:22.002630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:21.928012    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:22.232332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.225534    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:22.381632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.347710    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:22.446230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.364435    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:22.607695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.560860    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:31:22.607695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.560916    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:31:22.607695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.560970    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:31:22.802088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.802036    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"]
Oct 31 03:31:22.803246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.738115    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:22.957862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.957807    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:22.982640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.964580    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.019174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:22.990715    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.164127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.164078    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:23.404330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.404281    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.518656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.501635    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:23.518656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.501763    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:23.518656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.501811    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:23.588590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.584704    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.709670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.691873    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:31:23.721644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.720471    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:23.885685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.861967    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.979633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.969106    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:23.979633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:23.970279    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:24.001852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.001789    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:24.020671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.016950    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.020671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.017070    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.020671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.017240    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt6bf\" (UniqueName: \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.158585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.158480    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.224732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.193554    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:24.239671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.194767    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.271681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.242486    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.271681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.255096    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt6bf\" (UniqueName: \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.321657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.309587    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:24.408876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.408752    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:24.568634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.567057    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7264-189b9e" event={"ID":"9fb035da-b9b8-4a80-91c3-2eaca6672d83","Type":"ContainerStarted","Data":"ae26580cfaac4e71c9b0cea7602a969cd722aa917ea503ea6f2643e3485fd3ed"}
Oct 31 03:31:24.614749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.614709    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"]
Oct 31 03:31:24.629850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.596166    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:24.801845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.801790    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 31 03:31:24.883751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:24.883702    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.029828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.029724    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.066659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.064862    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.066659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.065272    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.093698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.091766    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:31:25.246650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.240541    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.424571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.382736    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:25.490689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.463482    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 03:31:25.490689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.463776    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.802766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.802731    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt6bf\" (UniqueName: \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\") pod \"maas-api-revoke-sub-3184a815-4dtgx\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:25.870942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.870899    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:25.961141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:25.892752    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:26.182246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.182198    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:26.365635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.357149    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:31:26.419844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.419773    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:31:26.420228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.420183    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:26.541717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.520768    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:26.757659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:26.638116    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:27.285042 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:27.285002    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:27.557639 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:27.557074    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.771s"
Oct 31 03:31:27.557639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:27.557234    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerStarted","Data":"0f0b517a31436b7d9b47c78aab0da347bf40be8873dc4e721ca305fe9d91ce6b"}
Oct 31 03:31:27.764692 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:27.759452    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98a7637d_8306_4f42_beb4_6452b1174e78.slice/crio-8f946fb7c78a913718083f60f01b3936b04861885e515e7a3e0ef3a7f540a933\": RecentStats: unable to find data in memory cache]"
Oct 31 03:31:28.141771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.141734    5098 generic.go:356] "Generic (PLEG): container finished" podID="9640791f-1a44-45f0-8931-218c90f0a671" containerID="7077d226c9852414b1386e9471b921cf5aa556a542dc375bd63b71fc5f068a29" exitCode=137
Oct 31 03:31:28.150592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.150554    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:31:28.155726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.155699    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:31:28.156001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.155983    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:31:28.156156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.156135    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerStarted","Data":"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"}
Oct 31 03:31:28.156305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.156289    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:31:28.156438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.156425    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:31:28.156576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.156564    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:31:28.156717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.156705    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:31:28.177363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.177331    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:28.215959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.212364    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:28.251299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.251193    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:28.281211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.281150    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podStartSLOduration=10.281130208 podStartE2EDuration="10.281130208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:28.275408991 +0000 UTC m=+962.800628582" watchObservedRunningTime="2026-10-31 03:31:28.281130208 +0000 UTC m=+962.806349798"
Oct 31 03:31:28.288685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.286007    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\") pod \"16e5fdec-290d-4826-9757-2282b078121c\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") "
Oct 31 03:31:28.288685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.286193    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\") pod \"16e5fdec-290d-4826-9757-2282b078121c\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") "
Oct 31 03:31:28.288685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.286258    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\" (UniqueName: \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\") pod \"16e5fdec-290d-4826-9757-2282b078121c\" (UID: \"16e5fdec-290d-4826-9757-2282b078121c\") "
Oct 31 03:31:28.289321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.289202    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca" pod "16e5fdec-290d-4826-9757-2282b078121c" (UID: "16e5fdec-290d-4826-9757-2282b078121c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:28.290257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.289887    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca" pod "16e5fdec-290d-4826-9757-2282b078121c" (UID: "16e5fdec-290d-4826-9757-2282b078121c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:28.306931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.306882    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q" pod "16e5fdec-290d-4826-9757-2282b078121c" (UID: "16e5fdec-290d-4826-9757-2282b078121c"). InnerVolumeSpecName "kube-api-access-49f4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:28.352576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.352490    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:28.387419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.387377    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\" (UniqueName: \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\") pod \"9640791f-1a44-45f0-8931-218c90f0a671\" (UID: \"9640791f-1a44-45f0-8931-218c90f0a671\") "
Oct 31 03:31:28.388413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.388384    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16e5fdec-290d-4826-9757-2282b078121c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:28.388639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.388596    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16e5fdec-290d-4826-9757-2282b078121c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:28.388639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.388642    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49f4q\" (UniqueName: \"kubernetes.io/projected/16e5fdec-290d-4826-9757-2282b078121c-kube-api-access-49f4q\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:28.391547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.391512    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb" pod "9640791f-1a44-45f0-8931-218c90f0a671" (UID: "9640791f-1a44-45f0-8931-218c90f0a671"). InnerVolumeSpecName "kube-api-access-msvnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:28.430369 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:28.430322    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-46ccf5\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9640791f-1a44-45f0-8931-218c90f0a671" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:28.504631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.500922    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msvnb\" (UniqueName: \"kubernetes.io/projected/9640791f-1a44-45f0-8931-218c90f0a671-kube-api-access-msvnb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:28.639736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:28.635134    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"]
Oct 31 03:31:29.165528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.165500    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"
Oct 31 03:31:29.165677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.165521    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd" event={"ID":"16e5fdec-290d-4826-9757-2282b078121c","Type":"ContainerDied","Data":"f34776f7e9dc678963d7e97dc9904fc7f314514693d4cc25ed6315c1e8a48500"}
Oct 31 03:31:29.165677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.165582    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34776f7e9dc678963d7e97dc9904fc7f314514693d4cc25ed6315c1e8a48500"
Oct 31 03:31:29.167263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.167241    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:29.167386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.167296    5098 scope.go:122] "RemoveContainer" containerID="7077d226c9852414b1386e9471b921cf5aa556a542dc375bd63b71fc5f068a29"
Oct 31 03:31:29.169790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.169758    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx" event={"ID":"b2c05363-ee13-4551-8c6f-18c748a06fc2","Type":"ContainerStarted","Data":"a1f0a6ebcaf9255e7f937fe08149ed143e2b568810a7bffc56c0fc2dc781688d"}
Oct 31 03:31:29.170125 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:29.169980    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-46ccf5\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9640791f-1a44-45f0-8931-218c90f0a671" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:29.174846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.174812    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7264-189b9e" event={"ID":"9fb035da-b9b8-4a80-91c3-2eaca6672d83","Type":"ContainerStarted","Data":"60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d"}
Oct 31 03:31:29.175243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.175219    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:29.176017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.175991    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:31:29.196017 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:29.195976    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-46ccf5\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9640791f-1a44-45f0-8931-218c90f0a671" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:29.196715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.196671    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7264-189b9e" podStartSLOduration=11.196657875 podStartE2EDuration="11.196657875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:29.192980815 +0000 UTC m=+963.718200405" watchObservedRunningTime="2026-10-31 03:31:29.196657875 +0000 UTC m=+963.721877465"
Oct 31 03:31:29.226615 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:29.226564    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-46ccf5\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9640791f-1a44-45f0-8931-218c90f0a671" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:29.266787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.266736    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:31:29.268042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.268009    5098 cpu_manager.go:414] "containerMap: removing container" podUID="16e5fdec-290d-4826-9757-2282b078121c" containerName="revoke-keys"
Oct 31 03:31:29.268042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.268040    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16e5fdec-290d-4826-9757-2282b078121c" containerName="revoke-keys"
Oct 31 03:31:29.268329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.268310    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="16e5fdec-290d-4826-9757-2282b078121c" containerName="revoke-keys"
Oct 31 03:31:29.274783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.274752    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.277254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.277217    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:31:29.281375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.281349    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5aeec060-dockercfg-pzjql\""
Oct 31 03:31:29.281375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.281349    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5aeec060\""
Oct 31 03:31:29.296307 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:29.296255    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-46ccf5\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9640791f-1a44-45f0-8931-218c90f0a671" pod="opendatahub/test-curl-7264-46ccf5"
Oct 31 03:31:29.440735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.434848    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.441161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.441076    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5qld\" (UniqueName: \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.549474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.549430    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.549817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.549793    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5qld\" (UniqueName: \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.561082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.550382    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.659629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.643681    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5qld\" (UniqueName: \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\") pod \"payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") " pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:29.698679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.698107    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9640791f-1a44-45f0-8931-218c90f0a671" path="/var/lib/kubelet/pods/9640791f-1a44-45f0-8931-218c90f0a671/volumes"
Oct 31 03:31:29.893142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:29.893109    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:30.191175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.191139    5098 generic.go:356] "Generic (PLEG): container finished" podID="b2c05363-ee13-4551-8c6f-18c748a06fc2" containerID="530de47fd8de70733d37c07a183ef6b58dbc6e23667348a430345c0694fcba50" exitCode=0
Oct 31 03:31:30.191336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.191177    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx" event={"ID":"b2c05363-ee13-4551-8c6f-18c748a06fc2","Type":"ContainerDied","Data":"530de47fd8de70733d37c07a183ef6b58dbc6e23667348a430345c0694fcba50"}
Oct 31 03:31:30.211952 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:30.211919    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f0389d_6a5c_4b9b_9e19_278c7c0ee73e.slice/crio-9b96ec9ca532994090d76e901df7f2c7aa549e4ed8a0443877f124585a500ea0 WatchSource:0}: Error finding container 9b96ec9ca532994090d76e901df7f2c7aa549e4ed8a0443877f124585a500ea0: Status 404 returned error can't find the container with id 9b96ec9ca532994090d76e901df7f2c7aa549e4ed8a0443877f124585a500ea0
Oct 31 03:31:30.227419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.227384    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:31:30.321785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.321743    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7264-189b9e"]
Oct 31 03:31:30.330554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.330515    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7264-189b9e"]
Oct 31 03:31:30.753141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.753102    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:31:30.753889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.753868    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" containerName="test-curl-7264-189b9e"
Oct 31 03:31:30.753943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.753892    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" containerName="test-curl-7264-189b9e"
Oct 31 03:31:30.754154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.754139    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" containerName="test-curl-7264-189b9e"
Oct 31 03:31:30.758310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.758289    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.770779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.770741    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-ed9442ec\""
Oct 31 03:31:30.770979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.770950    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-ed9442ec-dockercfg-x44jh\""
Oct 31 03:31:30.775140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.775114    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:31:30.804827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.804789    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jcb\" (UniqueName: \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.805043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.805019    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.813641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.813591    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:31:30.821024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.820993    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:30.827776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.827745    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:31:30.907403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.907360    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhtc\" (UniqueName: \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:30.907576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.907544    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.907700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.907680    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:30.907810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.907797    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jcb\" (UniqueName: \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.908898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.908878    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:30.927669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:30.920084    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jcb\" (UniqueName: \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\") pod \"payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:31.009525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.009428    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhtc\" (UniqueName: \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:31.009785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.009629    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:31.010400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.010380    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:31.018782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.018752    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhtc\" (UniqueName: \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\") pod \"payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") " pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:31.059596 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:31.059560    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:31:31.076154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.076111    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:31.144677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.143702    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:31.540429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.540358    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7264-189b9e" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" containerName="test-curl-7264-189b9e" containerID="cri-o://60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d" gracePeriod=2
Oct 31 03:31:31.543041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.540719    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" event={"ID":"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e","Type":"ContainerStarted","Data":"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"}
Oct 31 03:31:31.543389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.543361    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" event={"ID":"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e","Type":"ContainerStarted","Data":"9b96ec9ca532994090d76e901df7f2c7aa549e4ed8a0443877f124585a500ea0"}
Oct 31 03:31:31.543526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.543515    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:31.558445 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:31.552898    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-189b9e\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:31.601943 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:31.601888    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-189b9e\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:31.890336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.890237    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" podStartSLOduration=2.89021278 podStartE2EDuration="2.89021278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:31.59253691 +0000 UTC m=+966.117756554" watchObservedRunningTime="2026-10-31 03:31:31.89021278 +0000 UTC m=+966.415432467"
Oct 31 03:31:31.901773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.901750    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:31:31.911658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.911627    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:31:31.941592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:31.941556    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:32.133765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.133676    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\" (UniqueName: \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\") pod \"b2c05363-ee13-4551-8c6f-18c748a06fc2\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") "
Oct 31 03:31:32.133765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.133737    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\") pod \"b2c05363-ee13-4551-8c6f-18c748a06fc2\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") "
Oct 31 03:31:32.134030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.134014    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\") pod \"b2c05363-ee13-4551-8c6f-18c748a06fc2\" (UID: \"b2c05363-ee13-4551-8c6f-18c748a06fc2\") "
Oct 31 03:31:32.134425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.134395    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca" pod "b2c05363-ee13-4551-8c6f-18c748a06fc2" (UID: "b2c05363-ee13-4551-8c6f-18c748a06fc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:32.134758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.134740    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:32.135961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.135931    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf" pod "b2c05363-ee13-4551-8c6f-18c748a06fc2" (UID: "b2c05363-ee13-4551-8c6f-18c748a06fc2"). InnerVolumeSpecName "kube-api-access-pt6bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:32.136038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.135958    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca" pod "b2c05363-ee13-4551-8c6f-18c748a06fc2" (UID: "b2c05363-ee13-4551-8c6f-18c748a06fc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:32.235991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.235950    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt6bf\" (UniqueName: \"kubernetes.io/projected/b2c05363-ee13-4551-8c6f-18c748a06fc2-kube-api-access-pt6bf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:32.235991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.235985    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2c05363-ee13-4551-8c6f-18c748a06fc2-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:32.562229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.562197    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"
Oct 31 03:31:32.562489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.562459    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx" event={"ID":"b2c05363-ee13-4551-8c6f-18c748a06fc2","Type":"ContainerDied","Data":"a1f0a6ebcaf9255e7f937fe08149ed143e2b568810a7bffc56c0fc2dc781688d"}
Oct 31 03:31:32.562594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.562521    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1f0a6ebcaf9255e7f937fe08149ed143e2b568810a7bffc56c0fc2dc781688d"
Oct 31 03:31:32.567044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.567012    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" event={"ID":"4bb6d8b4-07fb-43a4-b45f-983178a8f11e","Type":"ContainerStarted","Data":"18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b"}
Oct 31 03:31:32.567210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.567065    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" event={"ID":"4bb6d8b4-07fb-43a4-b45f-983178a8f11e","Type":"ContainerStarted","Data":"bd986aec31cfc6e2be52a17483f070f83e252380452bbade7211ca4ec8873fb0"}
Oct 31 03:31:32.567210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.567093    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:32.569272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.569240    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" event={"ID":"90bb48bb-28f3-4733-89d5-86bf28f5ab05","Type":"ContainerStarted","Data":"4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6"}
Oct 31 03:31:32.569400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.569293    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" event={"ID":"90bb48bb-28f3-4733-89d5-86bf28f5ab05","Type":"ContainerStarted","Data":"726ebd0a3207620b880601c59e42986a52b9de0897cd646b96e2b836d19e0c9d"}
Oct 31 03:31:32.569400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.569330    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:32.593962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.593905    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" podStartSLOduration=2.59389043 podStartE2EDuration="2.59389043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:32.586769092 +0000 UTC m=+967.111988713" watchObservedRunningTime="2026-10-31 03:31:32.59389043 +0000 UTC m=+967.119110029"
Oct 31 03:31:32.615899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:32.615843    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" podStartSLOduration=2.6158280510000003 podStartE2EDuration="2.615828051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:32.611769797 +0000 UTC m=+967.136989385" watchObservedRunningTime="2026-10-31 03:31:32.615828051 +0000 UTC m=+967.141047639"
Oct 31 03:31:33.865752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:33.865721    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:33.915240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:33.915149    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\" (UniqueName: \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\") pod \"9fb035da-b9b8-4a80-91c3-2eaca6672d83\" (UID: \"9fb035da-b9b8-4a80-91c3-2eaca6672d83\") "
Oct 31 03:31:33.917419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:33.917389    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd" pod "9fb035da-b9b8-4a80-91c3-2eaca6672d83" (UID: "9fb035da-b9b8-4a80-91c3-2eaca6672d83"). InnerVolumeSpecName "kube-api-access-2cbjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:34.017509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.017474    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cbjd\" (UniqueName: \"kubernetes.io/projected/9fb035da-b9b8-4a80-91c3-2eaca6672d83-kube-api-access-2cbjd\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:34.670885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.664781    5098 generic.go:356] "Generic (PLEG): container finished" podID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" containerID="60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d" exitCode=137
Oct 31 03:31:34.670885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.665109    5098 scope.go:122] "RemoveContainer" containerID="60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d"
Oct 31 03:31:34.670885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.665288    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:34.711628 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:34.698828    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-189b9e\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:34.909677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.909075    5098 scope.go:122] "RemoveContainer" containerID="60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d"
Oct 31 03:31:34.909677 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:34.909528    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d\": container with ID starting with 60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d not found: ID does not exist" containerID="60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d"
Oct 31 03:31:34.909677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:34.909568    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d"} err="failed to get container status \"60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d\": rpc error: code = NotFound desc = could not find container \"60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d\": container with ID starting with 60f5eea3b95cfd7550557e9216ecc55d636379cef5adf21d55a49ea21e65be7d not found: ID does not exist"
Oct 31 03:31:35.613020 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:35.612980    5098 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7264-189b9e\" is forbidden: User \"system:node:ip-10-0-129-132.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-132.us-east-2.compute.internal' and this object" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" pod="opendatahub/test-curl-7264-189b9e"
Oct 31 03:31:35.614365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:35.614340    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb035da-b9b8-4a80-91c3-2eaca6672d83" path="/var/lib/kubelet/pods/9fb035da-b9b8-4a80-91c3-2eaca6672d83/volumes"
Oct 31 03:31:40.658179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.658107    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"]
Oct 31 03:31:40.660839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.659491    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b2c05363-ee13-4551-8c6f-18c748a06fc2" containerName="revoke-keys"
Oct 31 03:31:40.660839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.659558    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2c05363-ee13-4551-8c6f-18c748a06fc2" containerName="revoke-keys"
Oct 31 03:31:40.660839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.659835    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2c05363-ee13-4551-8c6f-18c748a06fc2" containerName="revoke-keys"
Oct 31 03:31:40.665457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.665425    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.670538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.670494    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:40.679392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.677146    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:40.679392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.677289    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"]
Oct 31 03:31:40.728904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.728420    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:31:40.728904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.728502    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:40.773883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.760874    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz2t\" (UniqueName: \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.821970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.821934    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.822581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.822549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.924913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.924818    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.925102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.924918    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.925102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.925079    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz2t\" (UniqueName: \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.925682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.925655    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.927379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.927355    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.934343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.934313    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz2t\" (UniqueName: \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\") pod \"maas-api-revoke-sub-3e83c26c-wgpxl\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:40.986964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:40.986919    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:41.447734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:41.427294    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"]
Oct 31 03:31:41.447734 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:41.434300    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060860f5_e3fd_41f7_92d7_86a31236be1d.slice/crio-c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c WatchSource:0}: Error finding container c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c: Status 404 returned error can't find the container with id c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c
Oct 31 03:31:41.912718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:41.912237    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl" event={"ID":"060860f5-e3fd-41f7-92d7-86a31236be1d","Type":"ContainerStarted","Data":"c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c"}
Oct 31 03:31:42.573332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:42.573302    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:31:42.926115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:42.925984    5098 generic.go:356] "Generic (PLEG): container finished" podID="060860f5-e3fd-41f7-92d7-86a31236be1d" containerID="6a5d27bedae344ea238c23290a5dad87caa2ed23d413eb49199f274d4a6de31a" exitCode=0
Oct 31 03:31:42.926115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:42.926064    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl" event={"ID":"060860f5-e3fd-41f7-92d7-86a31236be1d","Type":"ContainerDied","Data":"6a5d27bedae344ea238c23290a5dad87caa2ed23d413eb49199f274d4a6de31a"}
Oct 31 03:31:43.634036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:43.634005    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:31:43.634209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:43.634085    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:31:44.143520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.143491    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:44.245967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.245848    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\") pod \"060860f5-e3fd-41f7-92d7-86a31236be1d\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") "
Oct 31 03:31:44.246318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.245981    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\") pod \"060860f5-e3fd-41f7-92d7-86a31236be1d\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") "
Oct 31 03:31:44.246318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.246032    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\" (UniqueName: \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\") pod \"060860f5-e3fd-41f7-92d7-86a31236be1d\" (UID: \"060860f5-e3fd-41f7-92d7-86a31236be1d\") "
Oct 31 03:31:44.246318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.246242    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca" pod "060860f5-e3fd-41f7-92d7-86a31236be1d" (UID: "060860f5-e3fd-41f7-92d7-86a31236be1d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:44.247213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.247177    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:44.249122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.249088    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t" pod "060860f5-e3fd-41f7-92d7-86a31236be1d" (UID: "060860f5-e3fd-41f7-92d7-86a31236be1d"). InnerVolumeSpecName "kube-api-access-jkz2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:44.249239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.249145    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca" pod "060860f5-e3fd-41f7-92d7-86a31236be1d" (UID: "060860f5-e3fd-41f7-92d7-86a31236be1d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:44.371002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.363802    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060860f5-e3fd-41f7-92d7-86a31236be1d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:44.371002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:44.363942    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkz2t\" (UniqueName: \"kubernetes.io/projected/060860f5-e3fd-41f7-92d7-86a31236be1d-kube-api-access-jkz2t\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:45.016703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.016338    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl" event={"ID":"060860f5-e3fd-41f7-92d7-86a31236be1d","Type":"ContainerDied","Data":"c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c"}
Oct 31 03:31:45.016703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.016403    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5a80c679d9159b63d0c10e0e64c52f14b85b431688b3160b44e30a77396b25c"
Oct 31 03:31:45.016703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.016474    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"
Oct 31 03:31:45.042775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.042684    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"]
Oct 31 03:31:45.043868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.043835    5098 cpu_manager.go:414] "containerMap: removing container" podUID="060860f5-e3fd-41f7-92d7-86a31236be1d" containerName="revoke-keys"
Oct 31 03:31:45.043868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.043866    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060860f5-e3fd-41f7-92d7-86a31236be1d" containerName="revoke-keys"
Oct 31 03:31:45.044136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.044122    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="060860f5-e3fd-41f7-92d7-86a31236be1d" containerName="revoke-keys"
Oct 31 03:31:45.049083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.049054    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.050029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.050000    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"]
Oct 31 03:31:45.056664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.053847    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:31:45.056664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.053926    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:31:45.070234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.070203    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:31:45.070411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.070370    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:31:45.075438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.075403    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.075438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.075438    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltbwg\" (UniqueName: \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.075705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.075462    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.178053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.177993    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.178053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.178064    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltbwg\" (UniqueName: \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.178675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.178104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.179191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.179160    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.181027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.180992    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.187662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.187633    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltbwg\" (UniqueName: \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\") pod \"maas-api-revoke-sub-650abfd4-4sz6x\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.367849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.367773    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:45.867591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:45.862670    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"]
Oct 31 03:31:45.905291 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:45.905256    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284cf55f_8d82_4ba5_a804_85c66cb0c02c.slice/crio-52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8 WatchSource:0}: Error finding container 52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8: Status 404 returned error can't find the container with id 52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8
Oct 31 03:31:46.028472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:46.028370    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x" event={"ID":"284cf55f-8d82-4ba5-a804-85c66cb0c02c","Type":"ContainerStarted","Data":"52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8"}
Oct 31 03:31:47.048005 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.047962    5098 generic.go:356] "Generic (PLEG): container finished" podID="284cf55f-8d82-4ba5-a804-85c66cb0c02c" containerID="3597897894a8c837a72a1078ce3ec53941e8ac8bfc03ca86115753b207721840" exitCode=0
Oct 31 03:31:47.048377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.048208    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x" event={"ID":"284cf55f-8d82-4ba5-a804-85c66cb0c02c","Type":"ContainerDied","Data":"3597897894a8c837a72a1078ce3ec53941e8ac8bfc03ca86115753b207721840"}
Oct 31 03:31:47.496204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.496162    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"]
Oct 31 03:31:47.501634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.501583    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.512847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.512811    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"]
Oct 31 03:31:47.645353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.645309    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.645548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.645492    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5288c\" (UniqueName: \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.645651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.645618    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.756778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.752825    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5288c\" (UniqueName: \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.756778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.752981    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.756778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.753233    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.756778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.754845    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.772516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.771508    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.773306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.773280    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5288c\" (UniqueName: \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\") pod \"maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:47.822263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:47.822224    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:48.120805 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:48.120768    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf20f8e5d_6036_418b_b7a0_96ba4ca8bb57.slice/crio-2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc WatchSource:0}: Error finding container 2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc: Status 404 returned error can't find the container with id 2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc
Oct 31 03:31:48.124332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.124301    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"]
Oct 31 03:31:48.277510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.277477    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:48.363084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.362994    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\") pod \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") "
Oct 31 03:31:48.363084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.363046    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\" (UniqueName: \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\") pod \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") "
Oct 31 03:31:48.363355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.363338    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\") pod \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\" (UID: \"284cf55f-8d82-4ba5-a804-85c66cb0c02c\") "
Oct 31 03:31:48.363431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.363402    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca" pod "284cf55f-8d82-4ba5-a804-85c66cb0c02c" (UID: "284cf55f-8d82-4ba5-a804-85c66cb0c02c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:48.363967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.363935    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:48.368540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.368493    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca" pod "284cf55f-8d82-4ba5-a804-85c66cb0c02c" (UID: "284cf55f-8d82-4ba5-a804-85c66cb0c02c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:48.368540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.368527    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg" pod "284cf55f-8d82-4ba5-a804-85c66cb0c02c" (UID: "284cf55f-8d82-4ba5-a804-85c66cb0c02c"). InnerVolumeSpecName "kube-api-access-ltbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:48.465221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.465193    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/284cf55f-8d82-4ba5-a804-85c66cb0c02c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:48.465221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:48.465221    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltbwg\" (UniqueName: \"kubernetes.io/projected/284cf55f-8d82-4ba5-a804-85c66cb0c02c-kube-api-access-ltbwg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:49.071831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.071777    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2" event={"ID":"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57","Type":"ContainerStarted","Data":"bcd6b02424c6d95503b9484cc15e94363f48955f52d31384a4bb6630858d482e"}
Oct 31 03:31:49.072044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.071851    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2" event={"ID":"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57","Type":"ContainerStarted","Data":"2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc"}
Oct 31 03:31:49.073383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.073355    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"
Oct 31 03:31:49.073521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.073400    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x" event={"ID":"284cf55f-8d82-4ba5-a804-85c66cb0c02c","Type":"ContainerDied","Data":"52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8"}
Oct 31 03:31:49.073521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.073441    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52abb92f1c13f2e1d40884dc636b0bd3d53c0bb61eb17df7ab59ecf2e5c502b8"
Oct 31 03:31:49.123717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.123655    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2" podStartSLOduration=2.123635282 podStartE2EDuration="2.123635282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:49.116707447 +0000 UTC m=+983.641927039" watchObservedRunningTime="2026-10-31 03:31:49.123635282 +0000 UTC m=+983.648854872"
Oct 31 03:31:49.493016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.492913    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:31:49.496556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.495040    5098 cpu_manager.go:414] "containerMap: removing container" podUID="284cf55f-8d82-4ba5-a804-85c66cb0c02c" containerName="revoke-keys"
Oct 31 03:31:49.496556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.495069    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284cf55f-8d82-4ba5-a804-85c66cb0c02c" containerName="revoke-keys"
Oct 31 03:31:49.496556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.495585    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="284cf55f-8d82-4ba5-a804-85c66cb0c02c" containerName="revoke-keys"
Oct 31 03:31:49.502446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.502316    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.524923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.524887    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:31:49.531972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.531944    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.532439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.532408    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:31:49.541267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.541229    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:31:49.606697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606665    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.606697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606699    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.606932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606731    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.606932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606874    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.607077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606937    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.607077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.606990    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.607077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.607055    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.607185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.607085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.607260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.607232    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kfxv\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.630073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.630036    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"]
Oct 31 03:31:49.635900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.635860    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.650520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.650467    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"]
Oct 31 03:31:49.709068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709030    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kfxv\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709097    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.709258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709236    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lngdt\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.709402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709277    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709299    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709335    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.709585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709503    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709573    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.709742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709639    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709788    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709853    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.709977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709858    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.709977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.709962    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.710095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710023    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.710095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710073    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.710237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.710301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710261    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.710301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710272    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.710408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710347    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.710497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710475    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.710589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710573    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.710762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.710577    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.711235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.711211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.712539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.712516    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.713421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.713398    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.718129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.718105    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.718206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.718150    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kfxv\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.811990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.811948    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tggxs\" (UniqueName: \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.812185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812012    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812156    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.812313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812293    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812355    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lngdt\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812414    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812466    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812517    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812685    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812729    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.812804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812765    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812874    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812901    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.812969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.812951    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.813117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.813095    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.813244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.813223    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.813304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.813245    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.815255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.815234    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.815255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.815242    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.821303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.821274    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.821418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.821378    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lngdt\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\") pod \"e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") " pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.828331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.828304    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:49.852298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.852272    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:49.915157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.915055    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.915458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.915413    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.915573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.915492    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tggxs\" (UniqueName: \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.940349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.939536    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.943847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.943273    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.953188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.953155    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tggxs\" (UniqueName: \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\") pod \"maas-api-revoke-sub-ce35864d-7rrwc\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:49.965024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:49.964985    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:31:50.111585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.104301    5098 generic.go:356] "Generic (PLEG): container finished" podID="f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" containerID="bcd6b02424c6d95503b9484cc15e94363f48955f52d31384a4bb6630858d482e" exitCode=0
Oct 31 03:31:50.111585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.104759    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2" event={"ID":"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57","Type":"ContainerDied","Data":"bcd6b02424c6d95503b9484cc15e94363f48955f52d31384a4bb6630858d482e"}
Oct 31 03:31:50.159795 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:50.159749    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46cd7714_1095_4d02_80d3_fdd4c10724e5.slice/crio-bcbc66d9e5285cebde6254b66aee3463eb98e5de2e7433f6cacedc68363a083c WatchSource:0}: Error finding container bcbc66d9e5285cebde6254b66aee3463eb98e5de2e7433f6cacedc68363a083c: Status 404 returned error can't find the container with id bcbc66d9e5285cebde6254b66aee3463eb98e5de2e7433f6cacedc68363a083c
Oct 31 03:31:50.162977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.162308    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.162977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.162400    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.162977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.162432    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.168006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.167710    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:31:50.198986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.197876    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:31:50.204086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.204050    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.204294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.204279    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.204386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.204371    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:50.401042 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:31:50.400993    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode593357b_99cf_410f_bd56_61fcbbd4d177.slice/crio-f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb WatchSource:0}: Error finding container f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb: Status 404 returned error can't find the container with id f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb
Oct 31 03:31:50.401254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:50.401172    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"]
Oct 31 03:31:51.035652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.032571    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"]
Oct 31 03:31:51.050234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.048580    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"]
Oct 31 03:31:51.050234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.048741    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.090050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.090005    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.090312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.090085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.090312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.090123    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8wwj\" (UniqueName: \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.134923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.134875    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc" event={"ID":"e593357b-99cf-410f-bd56-61fcbbd4d177","Type":"ContainerStarted","Data":"23baa43d2c27c548e44ff2ea24207387fafcb7acaf29d785a548d7fa3f9d282d"}
Oct 31 03:31:51.135115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.134995    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc" event={"ID":"e593357b-99cf-410f-bd56-61fcbbd4d177","Type":"ContainerStarted","Data":"f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb"}
Oct 31 03:31:51.137302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.137267    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" event={"ID":"46cd7714-1095-4d02-80d3-fdd4c10724e5","Type":"ContainerStarted","Data":"df56cac72d640ecfc26027662f746e1a04acf30af824872172b8d6fe0f618a1b"}
Oct 31 03:31:51.137468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.137321    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" event={"ID":"46cd7714-1095-4d02-80d3-fdd4c10724e5","Type":"ContainerStarted","Data":"bcbc66d9e5285cebde6254b66aee3463eb98e5de2e7433f6cacedc68363a083c"}
Oct 31 03:31:51.139707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.139674    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" event={"ID":"673a2afe-eb7d-4de1-b1b6-24c8b2508485","Type":"ContainerStarted","Data":"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"}
Oct 31 03:31:51.139867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.139725    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" event={"ID":"673a2afe-eb7d-4de1-b1b6-24c8b2508485","Type":"ContainerStarted","Data":"c09aa343c81436d241d336fab4ff5700575da37a9a83d2611f799e0aa344020e"}
Oct 31 03:31:51.155422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.155339    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc" podStartSLOduration=2.155318409 podStartE2EDuration="2.155318409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:51.15275792 +0000 UTC m=+985.677977549" watchObservedRunningTime="2026-10-31 03:31:51.155318409 +0000 UTC m=+985.680538000"
Oct 31 03:31:51.179386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.179315    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podStartSLOduration=2.179292646 podStartE2EDuration="2.179292646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:51.176177773 +0000 UTC m=+985.701397381" watchObservedRunningTime="2026-10-31 03:31:51.179292646 +0000 UTC m=+985.704512236"
Oct 31 03:31:51.191497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.191455    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.191747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.191517    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.191747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.191554    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8wwj\" (UniqueName: \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.192556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.192520    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.196970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.196897    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.203485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.203457    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8wwj\" (UniqueName: \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\") pod \"maas-api-revoke-sub-d5ad200b-n27d4\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.208753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.208668    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podStartSLOduration=2.208637821 podStartE2EDuration="2.208637821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:51.201822983 +0000 UTC m=+985.727042572" watchObservedRunningTime="2026-10-31 03:31:51.208637821 +0000 UTC m=+985.733857410"
Oct 31 03:31:51.368722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.368684    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:51.375832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.375796    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:31:51.396508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.396471    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\" (UniqueName: \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\") pod \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") "
Oct 31 03:31:51.396729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.396591    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\") pod \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") "
Oct 31 03:31:51.398845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.396966    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\") pod \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\" (UID: \"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57\") "
Oct 31 03:31:51.398845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.398416    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca" pod "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" (UID: "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:31:51.431683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.402057    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca" pod "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" (UID: "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:31:51.431683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.405689    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c" pod "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" (UID: "f20f8e5d-6036-418b-b7a0-96ba4ca8bb57"). InnerVolumeSpecName "kube-api-access-5288c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:31:51.520982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.514430    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:51.520982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.514476    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5288c\" (UniqueName: \"kubernetes.io/projected/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-kube-api-access-5288c\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:51.520982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.514494    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:31:51.541630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.541141    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 03:31:51.544977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.542516    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" containerName="revoke-keys"
Oct 31 03:31:51.544977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.542546    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" containerName="revoke-keys"
Oct 31 03:31:51.544977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.542777    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" containerName="revoke-keys"
Oct 31 03:31:51.700631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.656853    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.764740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.702317    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.764740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.702503    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.764740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.702546    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.869647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.842894    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.869647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.842947    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.869647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.842981    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.869647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.843562    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7g8n\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.915134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.909197    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:51.915134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.909263    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:51.915134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.911592    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:51.915134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.911679    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:51.925329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.925296    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:31:51.925471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.925360    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:31:51.925931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.925879    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945514    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945555    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945629    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945664    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945685    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945784    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7g8n\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:51.958098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:51.945861    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.011641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.007667    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.095666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.053544    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7g8n\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.095666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.089057    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:31:52.259001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.258907    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.360707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.345227    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.429094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.429062    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.435339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.429597    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.435339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.430246    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qrrx8\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.435339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.433274    5098 generic.go:356] "Generic (PLEG): container finished" podID="e593357b-99cf-410f-bd56-61fcbbd4d177" containerID="23baa43d2c27c548e44ff2ea24207387fafcb7acaf29d785a548d7fa3f9d282d" exitCode=0
Oct 31 03:31:52.435339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.433412    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc" event={"ID":"e593357b-99cf-410f-bd56-61fcbbd4d177","Type":"ContainerDied","Data":"23baa43d2c27c548e44ff2ea24207387fafcb7acaf29d785a548d7fa3f9d282d"}
Oct 31 03:31:52.436535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.436503    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4" event={"ID":"a4598323-4668-4433-b1c2-516a7d390d59","Type":"ContainerStarted","Data":"e1ad56a92bd7cc32465f66f581d462d90928a24e3608b48a895b4707c389f05a"}
Oct 31 03:31:52.439723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.439687    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2" event={"ID":"f20f8e5d-6036-418b-b7a0-96ba4ca8bb57","Type":"ContainerDied","Data":"2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc"}
Oct 31 03:31:52.439877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.439731    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d3f10284b1e8f3ffd566ec268dbbd423487704f7a864c6731425d12a2e0d5bc"
Oct 31 03:31:52.439877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.439749    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"]
Oct 31 03:31:52.439877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.438787    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"
Oct 31 03:31:52.530036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.529993    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:31:52.838641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.830054    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:52.838641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.830122    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:52.916802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.885991    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:52.916802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:52.886064    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:54.175700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.151811    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:54.564336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.564232    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:54.596501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.596452    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:54.734421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.720811    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:54.734421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.731838    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:54.734421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.732912    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:54.790477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.743861    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:54.790477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.743945    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:54.790477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.755537    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:31:54.913633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.906746    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": context deadline exceeded"
Oct 31 03:31:54.939632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.912627    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:54.939826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.939759    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:54.939826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:54.939809    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:55.033634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.022808    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:55.033634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.022883    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:55.198640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.037318    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:55.222093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.063882    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.222093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.188336    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.222093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.188378    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:55.347634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.347143    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:31:55.451638 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:55.448382    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.812s"
Oct 31 03:31:55.451638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.448505    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"]
Oct 31 03:31:55.471580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.471546    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:31:55.521687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.484958    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:55.560385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.536530    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4" event={"ID":"a4598323-4668-4433-b1c2-516a7d390d59","Type":"ContainerStarted","Data":"9946afa99e45658ea3b8934e09a4ef50f8422df48ba0d3e96dae8a8d646df51c"}
Oct 31 03:31:55.734091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.734035    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.734752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.734716    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:55.870246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.843758    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.870556    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.870675    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.871659    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.871703    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.872188    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.872223    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:55.881633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.872281    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:55.970732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.906585    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:55.970732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.906960    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:55.970732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:55.907223    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b878\" (UniqueName: \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.048336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.048283    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b878\" (UniqueName: \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.162860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.162744    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.168044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.167992    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.186932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.186860    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.224539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.224477    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:56.263794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.263733    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:56.304398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.304247    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:31:56.564711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.546086    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b878\" (UniqueName: \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.698634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.663686    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-1c45857e-625gg\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.832729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.786171    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:56.832729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.787265    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:56.832729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.817321    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:31:56.876552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.876005    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:56.876552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.876453    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:31:56.981647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:56.937230    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:57.095658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.080748    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:57.095658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.080816    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:57.108630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.108498    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:57.108630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.108557    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:57.109196 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:31:57.109163    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.475s"
Oct 31 03:31:57.316292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.316262    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:31:57.336491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.336454    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:31:58.004012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.003906    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"]
Oct 31 03:31:58.284939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.284904    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:31:58.309635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.990738    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.310788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.046232    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4" podStartSLOduration=8.046207081 podStartE2EDuration="8.046207081s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:31:55.782253305 +0000 UTC m=+990.307472895" watchObservedRunningTime="2026-10-31 03:31:58.046207081 +0000 UTC m=+992.571426671"
Oct 31 03:31:58.342251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.046329    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.046356    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.077082    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.093734    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.093776    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.093799    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.097699    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.839802    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.389751    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:57.995297    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.390102    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.132833    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:31:58.409723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.390157    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:31:58.700959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.700908    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" event={"ID":"fe540f1a-81ec-4517-b816-fe5b54251eb1","Type":"ContainerStarted","Data":"a07fe787312955e5ddc4d986f08e67369e7445de056dfc442c435007b0530a5f"}
Oct 31 03:31:58.704779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.704734    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 03:31:58.708779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.708745    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:31:58.723792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.723751    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:31:58.748809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:58.748774    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:31:59.322058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:59.322006    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:31:59.474555 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:31:59.474508    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:00.006650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.002746    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:00.006650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.003127    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:00.006650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.003363    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:00.006650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.003667    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:00.182767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.178664    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:00.182767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.178734    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:00.410290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.410236    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:00.473692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.454543    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:00.473998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.473950    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:00.527181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.493681    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.61:50052: i/o timeout"
Oct 31 03:32:00.527181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.514848    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:00.527181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.514914    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:00.609629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.605986    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:00.609629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.606048    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:00.682854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.682809    5098 patch_prober.go:29] interesting pod/router-default-5497575d9c-lm8wr container/router namespace/openshift-ingress: Liveness probe status=failure output="Get \"http://10.132.0.17:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:00.683651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.683133    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/router-default-5497575d9c-lm8wr" podUID="b91cefe7-2e73-4f49-8d39-1187617dc044" containerName="router" probeResult="failure" output="Get \"http://10.132.0.17:1936/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:00.832117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.832049    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:00.832440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.832410    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:00.835288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.833987    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:00.835288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:00.834047    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.039638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.032706    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:32:01.087705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.064948    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:01.744724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.067873    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.744724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.122353    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:01.817429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.742664    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.817429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.296760    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.817429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.793832    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:01.830629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.820124    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.830629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.821078    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:01.859634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.388973    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:01.859634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.856799    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:01.859634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.478387    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:01.859634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:01.856941    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:02.016694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.013796    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.075634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.072756    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.173632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.127533    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.197641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.174247    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:02.327334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.316765    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.120:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.362671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.304039    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 31 03:32:02.362993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.274214    5098 patch_prober.go:29] interesting pod/jobset-controller-manager-8575547f5f-rqv7m container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:02.366628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.363111    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-8575547f5f-rqv7m" podUID="d3fa87e8-279f-4c28-ab71-2ed62d6a04ff" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.44:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.417628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.407314    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:02.417628 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:02.411167    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:32:02.505271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.505215    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 03:32:02.529220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.508459    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:32:02.790687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.786787    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:02.790687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.787216    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.010696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.911043    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.010696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.911492    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.010696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.911846    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.010696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:02.912144    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.090630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.090044    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.129658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.090583    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.129658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.094683    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:03.129658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.103534    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.154679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.130539    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\") pod \"e593357b-99cf-410f-bd56-61fcbbd4d177\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") "
Oct 31 03:32:03.154679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.130636    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\" (UniqueName: \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\") pod \"e593357b-99cf-410f-bd56-61fcbbd4d177\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") "
Oct 31 03:32:03.154679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.131018    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\") pod \"e593357b-99cf-410f-bd56-61fcbbd4d177\" (UID: \"e593357b-99cf-410f-bd56-61fcbbd4d177\") "
Oct 31 03:32:03.251878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.181899    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca" pod "e593357b-99cf-410f-bd56-61fcbbd4d177" (UID: "e593357b-99cf-410f-bd56-61fcbbd4d177"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:03.450760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.253773    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca" pod "e593357b-99cf-410f-bd56-61fcbbd4d177" (UID: "e593357b-99cf-410f-bd56-61fcbbd4d177"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:03.450760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.254092    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:03.450760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.254143    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:03.546882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.546845    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:03.686633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.685775    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.686633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.547663    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.686633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.598749    5098 patch_prober.go:29] interesting pod/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:03.686633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.686384    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg" podUID="3afa466d-1d28-4af9-a622-b3be1061dd07" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.808381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.761454    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:03.808381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.761521    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:03.808381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.800752    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 31 03:32:03.808381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.801221    5098 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.296178134s: [/var/lib/containers/storage/overlay/02cb6129652e91dded138e40a878e26774e2f8e6b8a52221d25f2de51ed2c205/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7_cab7e371-cb9c-46ca-be9c-faa8000ba33f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Oct 31 03:32:03.940641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.686591    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs" pod "e593357b-99cf-410f-bd56-61fcbbd4d177" (UID: "e593357b-99cf-410f-bd56-61fcbbd4d177"). InnerVolumeSpecName "kube-api-access-tggxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:03.965853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.915040    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:03.966148 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.966113    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:03.966292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.958470    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:03.991634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.930816    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.992235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.958827    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:03.996630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.930851    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:04.063656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.958880    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:04.064007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.063902    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:04.064230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:03.958794    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:04.128979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.128942    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:04.129258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.129235    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e593357b-99cf-410f-bd56-61fcbbd4d177-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:04.129421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.129405    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tggxs\" (UniqueName: \"kubernetes.io/projected/e593357b-99cf-410f-bd56-61fcbbd4d177-kube-api-access-tggxs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:04.146834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.146776    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:04.298275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.190224    5098 generic.go:356] "Generic (PLEG): container finished" podID="a4598323-4668-4433-b1c2-516a7d390d59" containerID="9946afa99e45658ea3b8934e09a4ef50f8422df48ba0d3e96dae8a8d646df51c" exitCode=0
Oct 31 03:32:04.298275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.242945    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:04.298275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.243021    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.135:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:04.339933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.334398    5098 patch_prober.go:29] interesting pod/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:04.339933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.334557    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:04.758688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.755273    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:04.810912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.801850    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:04.911039 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:04.911001    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="4.787s"
Oct 31 03:32:04.972754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.946257    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.000144    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.998014    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.980338    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.961407    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.000691    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.973115    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.000743    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:05.046906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:04.998050    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:05.177243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.080831    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:05.177243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.081226    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:05.195458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.195413    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:32:05.452036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.439890    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": context deadline exceeded"
Oct 31 03:32:05.487676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.486272    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534101    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" event={"ID":"fe540f1a-81ec-4517-b816-fe5b54251eb1","Type":"ContainerStarted","Data":"89d8e982c165a078acdeac0d885fb5a023f21d924a610618d7b150909eb4f43e"}
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534190    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4" event={"ID":"a4598323-4668-4433-b1c2-516a7d390d59","Type":"ContainerDied","Data":"9946afa99e45658ea3b8934e09a4ef50f8422df48ba0d3e96dae8a8d646df51c"}
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534305    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534346    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"]
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534377    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534427    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534450    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534486    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534504    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:32:05.554759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.534531    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:32:05.860649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.860351    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:05.903009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.869435    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:05.903009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.869499    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:05.986685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.986635    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:05.990465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:05.986931    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:06.111854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.111718    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:06.372509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.112088    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:06.548134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.548077    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:06.800406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.208023    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:06.814077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.208084    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:06.815412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.149791    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:06.900799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.900666    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" podStartSLOduration=15.900639428 podStartE2EDuration="15.900639428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:04.486901679 +0000 UTC m=+999.012121293" watchObservedRunningTime="2026-10-31 03:32:06.900639428 +0000 UTC m=+1001.425859018"
Oct 31 03:32:06.911228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.911169    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:06.915840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:06.915764    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:07.187695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.159319    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:07.401634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.386046    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:07.401634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.386107    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:07.401634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.393915    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:07.438727 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:07.432301    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.776s"
Oct 31 03:32:07.438727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.432542    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg" event={"ID":"7cce64ef-533c-4093-bf95-b464363bb168","Type":"ContainerStarted","Data":"96ff129dcb9d6a931e4413a9372c37b8ef0b8bb7ed7af76fb9d818c480ab0017"}
Oct 31 03:32:07.502784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.502741    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:07.503076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.503039    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:07.513819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.509581    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded"
Oct 31 03:32:07.515911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.515866    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"
Oct 31 03:32:07.522299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.521937    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc" event={"ID":"e593357b-99cf-410f-bd56-61fcbbd4d177","Type":"ContainerDied","Data":"f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb"}
Oct 31 03:32:07.525505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.525463    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3dc943370081dd2f3fc7dc4b0972b5011cf319f201ca9c06f2829973828e7fb"
Oct 31 03:32:07.563104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.563061    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-b8m6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:07.687879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.673552    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-b8m6d" podUID="aee37c72-f7bb-459c-9f06-df7401cc1daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:07.819659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.808083    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:07.819659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.809307    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:07.819659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.809379    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:07.856632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.854935    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:07.856632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.855405    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:07.882632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.860344    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:08.049868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.030129    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:08.049868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.914873    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:08.049868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.914833    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:32:08.049868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.914968    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:08.060254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:07.914992    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:08.060683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.027697    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:08.272368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.272321    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"]
Oct 31 03:32:08.307193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.307070    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:08.466630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.439895    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:08.634803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.634696    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5aeec060-k6pd2"]
Oct 31 03:32:08.765389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.765307    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg" event={"ID":"7cce64ef-533c-4093-bf95-b464363bb168","Type":"ContainerStarted","Data":"d81cbcb2b335ce249017a1735310eac00e81d70a668ad222966c752bbcfd2503"}
Oct 31 03:32:08.812132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.812088    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:08.876239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.850548    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:08.879322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.879277    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:08.879666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.864955    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:08.879779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:08.879704    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:09.048639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.044731    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:09.048639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.045629    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:32:09.090685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.067673    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:09.129799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.129738    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:09.323632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.312736    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:09.465698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.444438    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:09.746638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.746526    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f20f8e5d-6036-418b-b7a0-96ba4ca8bb57" path="/var/lib/kubelet/pods/f20f8e5d-6036-418b-b7a0-96ba4ca8bb57/volumes"
Oct 31 03:32:09.747509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.747466    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:32:09.846229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.845802    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:09.846229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.845874    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:09.859521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.852809    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:32:09.975641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.968009    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:09.975641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:09.968077    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:10.058824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.058768    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:10.079110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.078707    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:10.270686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.263941    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:10.445633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.429902    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg" podStartSLOduration=17.429877842 podStartE2EDuration="17.429877842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:31:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:08.838108988 +0000 UTC m=+1003.363328577" watchObservedRunningTime="2026-10-31 03:32:10.429877842 +0000 UTC m=+1004.955097432"
Oct 31 03:32:10.470593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.466410    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:10.490080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.487071    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:32:10.507473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.506448    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:10.541658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.538711    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" podUID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" containerName="payload-processing" containerID="cri-o://bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7" gracePeriod=25
Oct 31 03:32:10.744635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.737186    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:32:10.800739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.738421    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:32:10.903632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.903386    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:10.903632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.903478    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:10.927639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.914395    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\" (UniqueName: \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\") pod \"a4598323-4668-4433-b1c2-516a7d390d59\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") "
Oct 31 03:32:10.927639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.914500    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\") pod \"a4598323-4668-4433-b1c2-516a7d390d59\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") "
Oct 31 03:32:10.927639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.914775    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\") pod \"a4598323-4668-4433-b1c2-516a7d390d59\" (UID: \"a4598323-4668-4433-b1c2-516a7d390d59\") "
Oct 31 03:32:10.928564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.928196    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca" pod "a4598323-4668-4433-b1c2-516a7d390d59" (UID: "a4598323-4668-4433-b1c2-516a7d390d59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:10.964706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:10.951624    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj" pod "a4598323-4668-4433-b1c2-516a7d390d59" (UID: "a4598323-4668-4433-b1c2-516a7d390d59"). InnerVolumeSpecName "kube-api-access-c8wwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:11.032658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.027141    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca" pod "a4598323-4668-4433-b1c2-516a7d390d59" (UID: "a4598323-4668-4433-b1c2-516a7d390d59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:11.032868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.032727    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:32:11.072632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.035648    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:11.072632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.035706    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:11.072632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.061717    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:11.171641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.169358    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:11.290805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.217412    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:11.370641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.354393    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:11.370641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.354438    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8wwj\" (UniqueName: \"kubernetes.io/projected/a4598323-4668-4433-b1c2-516a7d390d59-kube-api-access-c8wwj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:11.370641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.354454    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4598323-4668-4433-b1c2-516a7d390d59-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:11.521957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.521921    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"]
Oct 31 03:32:11.526691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.526660    5098 cpu_manager.go:414] "containerMap: removing container" podUID="a4598323-4668-4433-b1c2-516a7d390d59" containerName="revoke-keys"
Oct 31 03:32:11.526952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.526931    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4598323-4668-4433-b1c2-516a7d390d59" containerName="revoke-keys"
Oct 31 03:32:11.527233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.527215    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e593357b-99cf-410f-bd56-61fcbbd4d177" containerName="revoke-keys"
Oct 31 03:32:11.539102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.538686    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e593357b-99cf-410f-bd56-61fcbbd4d177" containerName="revoke-keys"
Oct 31 03:32:11.539718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.539696    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e593357b-99cf-410f-bd56-61fcbbd4d177" containerName="revoke-keys"
Oct 31 03:32:11.540011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.539998    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4598323-4668-4433-b1c2-516a7d390d59" containerName="revoke-keys"
Oct 31 03:32:11.572299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.571474    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:11.636174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.616409    5098 generic.go:356] "Generic (PLEG): container finished" podID="7cce64ef-533c-4093-bf95-b464363bb168" containerID="d81cbcb2b335ce249017a1735310eac00e81d70a668ad222966c752bbcfd2503" exitCode=0
Oct 31 03:32:11.637809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.637770    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg" event={"ID":"7cce64ef-533c-4093-bf95-b464363bb168","Type":"ContainerDied","Data":"d81cbcb2b335ce249017a1735310eac00e81d70a668ad222966c752bbcfd2503"}
Oct 31 03:32:11.638054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.638036    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:32:11.641110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.641082    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:11.744046 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:11.743966    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f0389d_6a5c_4b9b_9e19_278c7c0ee73e.slice/crio-bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:32:11.860854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.860754    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:11.861124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.861087    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:11.938471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.938431    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:32:11.939567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.938793    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"]
Oct 31 03:32:11.944035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.942851    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:11.944035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.942971    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:11.944035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.943061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj852\" (UniqueName: \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:11.944332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.944275    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:11.944463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:11.944396    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:12.031316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.031278    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:32:12.044928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.044887    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\") pod \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") "
Oct 31 03:32:12.045123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.045100    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\" (UniqueName: \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\") pod \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\" (UID: \"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e\") "
Oct 31 03:32:12.045401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.045377    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.045499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.045482    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.045586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.045572    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj852\" (UniqueName: \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.045879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.045830    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume" pod "f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" (UID: "f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:12.047456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.047428    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.192004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.191807    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:12.208897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.208853    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld" pod "f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" (UID: "f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e"). InnerVolumeSpecName "kube-api-access-n5qld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:12.320887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.320849    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5qld\" (UniqueName: \"kubernetes.io/projected/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e-kube-api-access-n5qld\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:12.336726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.336691    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.393550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.373306    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj852\" (UniqueName: \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\") pod \"maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.448105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.448021    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:12.791454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.791416    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4" event={"ID":"a4598323-4668-4433-b1c2-516a7d390d59","Type":"ContainerDied","Data":"e1ad56a92bd7cc32465f66f581d462d90928a24e3608b48a895b4707c389f05a"}
Oct 31 03:32:12.792853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.792817    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1ad56a92bd7cc32465f66f581d462d90928a24e3608b48a895b4707c389f05a"
Oct 31 03:32:12.793149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.793135    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"
Oct 31 03:32:12.803990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.803952    5098 generic.go:356] "Generic (PLEG): container finished" podID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" containerID="bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7" exitCode=0
Oct 31 03:32:12.804626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.803993    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"
Oct 31 03:32:12.816299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.804043    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" event={"ID":"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e","Type":"ContainerDied","Data":"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"}
Oct 31 03:32:12.816299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.808166    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v" event={"ID":"f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e","Type":"ContainerDied","Data":"9b96ec9ca532994090d76e901df7f2c7aa549e4ed8a0443877f124585a500ea0"}
Oct 31 03:32:12.816299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.808210    5098 scope.go:122] "RemoveContainer" containerID="bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"
Oct 31 03:32:12.833295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.833260    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body=
Oct 31 03:32:12.850682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.850635    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused"
Oct 31 03:32:12.857094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.854406    5098 patch_prober.go:29] interesting pod/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 03:32:12.857094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.854461    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 03:32:12.862471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.862073    5098 scope.go:122] "RemoveContainer" containerID="bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"
Oct 31 03:32:12.863086 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:12.862911    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7\": container with ID starting with bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7 not found: ID does not exist" containerID="bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"
Oct 31 03:32:12.863086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.862952    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7"} err="failed to get container status \"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7\": rpc error: code = NotFound desc = could not find container \"bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7\": container with ID starting with bb049579cad96a35f13f2737bc7b827379bd576f27da1480ad2b4f42fbf317e7 not found: ID does not exist"
Oct 31 03:32:12.913020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.911695    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:32:12.928676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:12.923151    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5aeec060-d7f466bc7-2222v"]
Oct 31 03:32:13.138984 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:32:13.138898    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb65c530_2c74_47dd_a9d4_0b863130c22c.slice/crio-156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408 WatchSource:0}: Error finding container 156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408: Status 404 returned error can't find the container with id 156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408
Oct 31 03:32:13.155758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.155232    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"]
Oct 31 03:32:13.157697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.157460    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:32:13.179181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.175909    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\") pod \"7cce64ef-533c-4093-bf95-b464363bb168\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") "
Oct 31 03:32:13.179181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.176039    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\") pod \"7cce64ef-533c-4093-bf95-b464363bb168\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") "
Oct 31 03:32:13.179181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.176159    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\" (UniqueName: \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\") pod \"7cce64ef-533c-4093-bf95-b464363bb168\" (UID: \"7cce64ef-533c-4093-bf95-b464363bb168\") "
Oct 31 03:32:13.179181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.178071    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca" pod "7cce64ef-533c-4093-bf95-b464363bb168" (UID: "7cce64ef-533c-4093-bf95-b464363bb168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:13.180528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.180489    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878" pod "7cce64ef-533c-4093-bf95-b464363bb168" (UID: "7cce64ef-533c-4093-bf95-b464363bb168"). InnerVolumeSpecName "kube-api-access-8b878". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:13.182064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.182041    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca" pod "7cce64ef-533c-4093-bf95-b464363bb168" (UID: "7cce64ef-533c-4093-bf95-b464363bb168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:13.279951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.279797    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:13.279951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.279838    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cce64ef-533c-4093-bf95-b464363bb168-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:13.279951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.279854    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b878\" (UniqueName: \"kubernetes.io/projected/7cce64ef-533c-4093-bf95-b464363bb168-kube-api-access-8b878\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:13.654944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.654898    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" path="/var/lib/kubelet/pods/f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e/volumes"
Oct 31 03:32:13.910377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.910253    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8" event={"ID":"eb65c530-2c74-47dd-a9d4-0b863130c22c","Type":"ContainerStarted","Data":"504d067eb4f327920d62a8500f8a1e6e7b644aeb4e61af39b64098d9afb7b0b9"}
Oct 31 03:32:13.910651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.910627    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8" event={"ID":"eb65c530-2c74-47dd-a9d4-0b863130c22c","Type":"ContainerStarted","Data":"156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408"}
Oct 31 03:32:13.915456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.915421    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg" event={"ID":"7cce64ef-533c-4093-bf95-b464363bb168","Type":"ContainerDied","Data":"96ff129dcb9d6a931e4413a9372c37b8ef0b8bb7ed7af76fb9d818c480ab0017"}
Oct 31 03:32:13.915798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.915778    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ff129dcb9d6a931e4413a9372c37b8ef0b8bb7ed7af76fb9d818c480ab0017"
Oct 31 03:32:13.915949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.915714    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"
Oct 31 03:32:13.939208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.939060    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:32:13.940996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.940970    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:32:13.955979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.955943    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:32:13.958050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.958006    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:32:13.980756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:13.980687    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8" podStartSLOduration=3.980666209 podStartE2EDuration="3.980666209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:32:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:32:13.948497137 +0000 UTC m=+1008.473716726" watchObservedRunningTime="2026-10-31 03:32:13.980666209 +0000 UTC m=+1008.505885801"
Oct 31 03:32:14.929621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:14.929563    5098 generic.go:356] "Generic (PLEG): container finished" podID="eb65c530-2c74-47dd-a9d4-0b863130c22c" containerID="504d067eb4f327920d62a8500f8a1e6e7b644aeb4e61af39b64098d9afb7b0b9" exitCode=0
Oct 31 03:32:14.930085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:14.929672    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8" event={"ID":"eb65c530-2c74-47dd-a9d4-0b863130c22c","Type":"ContainerDied","Data":"504d067eb4f327920d62a8500f8a1e6e7b644aeb4e61af39b64098d9afb7b0b9"}
Oct 31 03:32:15.232687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:15.228736    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:32:15.232687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:15.228993    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:16.260805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.260722    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:16.402300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.402257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\" (UniqueName: \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\") pod \"eb65c530-2c74-47dd-a9d4-0b863130c22c\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") "
Oct 31 03:32:16.402729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.402700    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\") pod \"eb65c530-2c74-47dd-a9d4-0b863130c22c\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") "
Oct 31 03:32:16.402839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.402787    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\") pod \"eb65c530-2c74-47dd-a9d4-0b863130c22c\" (UID: \"eb65c530-2c74-47dd-a9d4-0b863130c22c\") "
Oct 31 03:32:16.403120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.403093    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca" pod "eb65c530-2c74-47dd-a9d4-0b863130c22c" (UID: "eb65c530-2c74-47dd-a9d4-0b863130c22c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:16.403688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.403663    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:16.404772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.404744    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852" pod "eb65c530-2c74-47dd-a9d4-0b863130c22c" (UID: "eb65c530-2c74-47dd-a9d4-0b863130c22c"). InnerVolumeSpecName "kube-api-access-bj852". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:16.404961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.404938    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca" pod "eb65c530-2c74-47dd-a9d4-0b863130c22c" (UID: "eb65c530-2c74-47dd-a9d4-0b863130c22c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:16.506088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.505992    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb65c530-2c74-47dd-a9d4-0b863130c22c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:16.506088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.506039    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj852\" (UniqueName: \"kubernetes.io/projected/eb65c530-2c74-47dd-a9d4-0b863130c22c-kube-api-access-bj852\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:16.952465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.952431    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"
Oct 31 03:32:16.952465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.952454    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8" event={"ID":"eb65c530-2c74-47dd-a9d4-0b863130c22c","Type":"ContainerDied","Data":"156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408"}
Oct 31 03:32:16.952751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:16.952515    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="156f0f29943f1ae155ba190d786e33ab46841e4b1729fdf75988991a72724408"
Oct 31 03:32:19.998111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:19.998075    5098 generic.go:356] "Generic (PLEG): container finished" podID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerID="5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116" exitCode=0
Oct 31 03:32:19.998567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:19.998145    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerDied","Data":"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"}
Oct 31 03:32:19.998803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:19.998782    5098 scope.go:122] "RemoveContainer" containerID="5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"
Oct 31 03:32:20.000520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:20.000491    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:20.000588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:20.000568    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:20.000688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:20.000672    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:32:21.017630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.010719    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerStarted","Data":"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"}
Oct 31 03:32:21.110630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.109657    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:32:21.212631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.212161    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:32:21.212631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.212191    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:21.212631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.212245    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:21.237310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:21.237051    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:32:22.223632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:22.217758    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:22.223632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:22.217898    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:22.748431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:22.748381    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:22.793783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:22.793711    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:23.017347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.017242    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:23.121772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.120711    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:23.210112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.210067    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:23.210401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.210368    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:23.401012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.400965    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:23.402623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.401584    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:23.498631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.495439    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:23.758700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.757715    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:23.758700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.758387    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:32:23.807671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.804455    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:23.807671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:23.805122    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:32:24.095951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.041468    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:24.095951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.042107    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:32:24.186634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.153514    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:24.579305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.337251    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:24.624795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.624736    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:24.625088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.455549    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:24.625538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.455509    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:24.626812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.521789    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:24.676634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.580349    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:32:24.813789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.813746    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:32:24.931987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.873544    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:32:24.931987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.873623    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:32:24.931987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.734347    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:24.931987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.769213    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:24.931987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:24.825070    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:25.064634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.063971    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:25.239691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.233817    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:25.239691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.233884    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:25.453773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.453722    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:25.454291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.454259    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": context deadline exceeded"
Oct 31 03:32:25.599909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.599860    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:25.668641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.663705    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:25.735641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.683623    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:25.735641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:25.684698    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:26.262698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.066754    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:26.300640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.266916    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" podUID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" containerName="payload-processing" containerID="cri-o://4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6" gracePeriod=25
Oct 31 03:32:26.500631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.323730    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:26.500631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.444755    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" podUID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" containerName="payload-pre-processing" containerID="cri-o://18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b" gracePeriod=25
Oct 31 03:32:26.731811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.731579    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:26.804249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:26.741678    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:27.113828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.084055    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:27.263795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.254081    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:27.398333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.398267    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:27.792044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.791985    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:27.814004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.813939    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:27.903626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.899475    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:27.903626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:27.899548    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:28.046687 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:28.034165    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.414s"
Oct 31 03:32:28.505933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.505835    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:28.506228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.506177    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:28.506858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.506818    5098 generic.go:356] "Generic (PLEG): container finished" podID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" containerID="18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b" exitCode=0
Oct 31 03:32:28.728664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.720961    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:28.728664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.723379    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:28.729116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.729077    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:32:28.903991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.903949    5098 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-nq5jp container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.11:8443/readyz\": context deadline exceeded" start-of-body=
Oct 31 03:32:28.904660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.904620    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/readyz\": context deadline exceeded"
Oct 31 03:32:28.904840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.851334    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 31 03:32:28.904979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.851377    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:28.906026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.729426    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:28.969947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.953539    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:28.969947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.953660    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded"
Oct 31 03:32:28.969947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.956946    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:29.307657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:29.302570    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:29.713877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:29.707740    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:29.713877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:29.708260    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:29.713877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:29.708312    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:30.216760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.216714    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:30.243803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.243747    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:30.267119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:28.729363    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:30.444435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.422534    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:30.444435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.422590    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:30.542664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.539409    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:30.614631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.566310    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:32:30.614631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.584387    5098 generic.go:356] "Generic (PLEG): container finished" podID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" containerID="4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6" exitCode=0
Oct 31 03:32:30.614631 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:30.588637    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.554s"
Oct 31 03:32:30.614631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.605756    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" event={"ID":"4bb6d8b4-07fb-43a4-b45f-983178a8f11e","Type":"ContainerDied","Data":"18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b"}
Oct 31 03:32:30.711107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.711060    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:30.843754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.711559    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:30.913309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.913276    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:32:30.957205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:30.929882    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:32:31.022020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.008803    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\" (UniqueName: \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\") pod \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") "
Oct 31 03:32:31.022020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.008886    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\" (UniqueName: \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\") pod \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") "
Oct 31 03:32:31.022020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.009116    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\") pod \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\" (UID: \"90bb48bb-28f3-4733-89d5-86bf28f5ab05\") "
Oct 31 03:32:31.022020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.009178    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\") pod \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\" (UID: \"4bb6d8b4-07fb-43a4-b45f-983178a8f11e\") "
Oct 31 03:32:31.023143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023108    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume" pod "90bb48bb-28f3-4733-89d5-86bf28f5ab05" (UID: "90bb48bb-28f3-4733-89d5-86bf28f5ab05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:31.023342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023237    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"]
Oct 31 03:32:31.023500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023474    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" event={"ID":"90bb48bb-28f3-4733-89d5-86bf28f5ab05","Type":"ContainerDied","Data":"4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6"}
Oct 31 03:32:31.023694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023677    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:32:31.023900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023886    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:32:31.025485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.025466    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:32:31.027101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.023803    5098 scope.go:122] "RemoveContainer" containerID="4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6"
Oct 31 03:32:31.035524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.024590    5098 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" secret="" err="secret \"e2e-migrate-ed9442ec-gw-openshift-default-dockercfg-w8ldd\" not found"
Oct 31 03:32:31.037568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.037535    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume" pod "4bb6d8b4-07fb-43a4-b45f-983178a8f11e" (UID: "4bb6d8b4-07fb-43a4-b45f-983178a8f11e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:31.071012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.070972    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc" pod "90bb48bb-28f3-4733-89d5-86bf28f5ab05" (UID: "90bb48bb-28f3-4733-89d5-86bf28f5ab05"). InnerVolumeSpecName "kube-api-access-zlhtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:31.102303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.102200    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb" pod "4bb6d8b4-07fb-43a4-b45f-983178a8f11e" (UID: "4bb6d8b4-07fb-43a4-b45f-983178a8f11e"). InnerVolumeSpecName "kube-api-access-d7jcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:31.191745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.185715    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:31.209293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.209107    5098 patch_prober.go:29] interesting pod/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 31 03:32:31.209578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.209541    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 31 03:32:31.212872 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.212838    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90bb48bb-28f3-4733-89d5-86bf28f5ab05-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:31.213058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.213040    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:31.213181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.213167    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlhtc\" (UniqueName: \"kubernetes.io/projected/90bb48bb-28f3-4733-89d5-86bf28f5ab05-kube-api-access-zlhtc\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:31.213277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.213265    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jcb\" (UniqueName: \"kubernetes.io/projected/4bb6d8b4-07fb-43a4-b45f-983178a8f11e-kube-api-access-d7jcb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:31.250948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.250894    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:31.357791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.357692    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:32:31.381840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.358944    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:32:31.435637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.406538    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ed9442ec-zjtf8"]
Oct 31 03:32:31.621866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.621742    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:31.883031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.882921    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:32:31.918691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:31.887999    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:32:32.020769 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:32.020714    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb6d8b4_07fb_43a4_b45f_983178a8f11e.slice/crio-18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb6d8b4_07fb_43a4_b45f_983178a8f11e.slice/crio-conmon-18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb48bb_28f3_4733_89d5_86bf28f5ab05.slice/crio-4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90bb48bb_28f3_4733_89d5_86bf28f5ab05.slice/crio-conmon-4296bf0a5bb45527ea49e42fd52c7fee8335cdc623870819b6bf655f9d74c7a6.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:32:32.042373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.038034    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb65c530-2c74-47dd-a9d4-0b863130c22c" path="/var/lib/kubelet/pods/eb65c530-2c74-47dd-a9d4-0b863130c22c/volumes"
Oct 31 03:32:32.064543 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:32.064489    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:32:32.072901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.072869    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:32:32.073238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.073222    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:32:32.073429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.073417    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:32:32.087246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.087137    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:32:32.110734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.110547    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d" event={"ID":"4bb6d8b4-07fb-43a4-b45f-983178a8f11e","Type":"ContainerDied","Data":"bd986aec31cfc6e2be52a17483f070f83e252380452bbade7211ca4ec8873fb0"}
Oct 31 03:32:32.110927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.110777    5098 scope.go:122] "RemoveContainer" containerID="18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b"
Oct 31 03:32:32.111735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.111714    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"
Oct 31 03:32:32.112634 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:32.112584    5098 kuberuntime_gc.go:392] "Failed to remove container log dead symlink" err="remove /var/log/containers/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d_openshift-ingress_payload-pre-processing-18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b.log: no such file or directory" path="/var/log/containers/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d_openshift-ingress_payload-pre-processing-18cd6e455bc0e1ea27722a1bfe0624617f5fbefcdd2b1d9bc450b46b77c8db1b.log"
Oct 31 03:32:32.116910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.114894    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"
Oct 31 03:32:32.116910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.115198    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb" event={"ID":"90bb48bb-28f3-4733-89d5-86bf28f5ab05","Type":"ContainerDied","Data":"726ebd0a3207620b880601c59e42986a52b9de0897cd646b96e2b836d19e0c9d"}
Oct 31 03:32:32.116910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.115825    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy" containerID="cri-o://8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985" gracePeriod=30
Oct 31 03:32:32.413006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.412965    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:32:32.423706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.423666    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-ed9442ec-5566647f45-n6lwb"]
Oct 31 03:32:32.431273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.431242    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:32:32.438738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.438706    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-ed9442ec-56d7786875-zg76d"]
Oct 31 03:32:32.892706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.882653    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"]
Oct 31 03:32:32.929630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:32.928747    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-1c45857e-625gg"]
Oct 31 03:32:33.642354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:33.640531    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" path="/var/lib/kubelet/pods/4bb6d8b4-07fb-43a4-b45f-983178a8f11e/volumes"
Oct 31 03:32:33.642354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:33.641299    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cce64ef-533c-4093-bf95-b464363bb168" path="/var/lib/kubelet/pods/7cce64ef-533c-4093-bf95-b464363bb168/volumes"
Oct 31 03:32:33.642354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:33.641821    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" path="/var/lib/kubelet/pods/90bb48bb-28f3-4733-89d5-86bf28f5ab05/volumes"
Oct 31 03:32:35.041685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.041424    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"]
Oct 31 03:32:35.042811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042784    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" containerName="payload-processing"
Oct 31 03:32:35.042811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042808    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" containerName="payload-processing"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042822    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7cce64ef-533c-4093-bf95-b464363bb168" containerName="revoke-keys"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042830    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cce64ef-533c-4093-bf95-b464363bb168" containerName="revoke-keys"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042914    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" containerName="payload-pre-processing"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042921    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" containerName="payload-pre-processing"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042936    5098 cpu_manager.go:414] "containerMap: removing container" podUID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" containerName="payload-processing"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042943    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" containerName="payload-processing"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042954    5098 cpu_manager.go:414] "containerMap: removing container" podUID="eb65c530-2c74-47dd-a9d4-0b863130c22c" containerName="revoke-keys"
Oct 31 03:32:35.043013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.042960    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb65c530-2c74-47dd-a9d4-0b863130c22c" containerName="revoke-keys"
Oct 31 03:32:35.043258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.043128    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3f0389d-6a5c-4b9b-9e19-278c7c0ee73e" containerName="payload-processing"
Oct 31 03:32:35.043258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.043142    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cce64ef-533c-4093-bf95-b464363bb168" containerName="revoke-keys"
Oct 31 03:32:35.043258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.043163    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb6d8b4-07fb-43a4-b45f-983178a8f11e" containerName="payload-pre-processing"
Oct 31 03:32:35.043258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.043174    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="90bb48bb-28f3-4733-89d5-86bf28f5ab05" containerName="payload-processing"
Oct 31 03:32:35.043258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.043181    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb65c530-2c74-47dd-a9d4-0b863130c22c" containerName="revoke-keys"
Oct 31 03:32:35.050486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.050439    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.054372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.054317    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"]
Oct 31 03:32:35.058800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.058771    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:32:35.060191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.060167    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:32:35.066220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.066190    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:32:35.066353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.066335    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:32:35.278575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.278527    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.278802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.278594    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwqqf\" (UniqueName: \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.278802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.278743    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.381087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.380819    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.381297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.381088    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwqqf\" (UniqueName: \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.381297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.381173    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.381962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.381933    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.383948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.383919    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.391514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.391480    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwqqf\" (UniqueName: \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\") pod \"maas-api-revoke-keys-e2e-derive-72b46d88-qgk22\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.671946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:35.671862    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:35.998941 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:32:35.998894    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ccf889_9140_4bbb_9610_4af464272471.slice/crio-ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e WatchSource:0}: Error finding container ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e: Status 404 returned error can't find the container with id ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e
Oct 31 03:32:36.001237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:36.001207    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"]
Oct 31 03:32:36.333395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:36.333354    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22" event={"ID":"d2ccf889-9140-4bbb-9610-4af464272471","Type":"ContainerStarted","Data":"ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e"}
Oct 31 03:32:37.345738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:37.345698    5098 generic.go:356] "Generic (PLEG): container finished" podID="d2ccf889-9140-4bbb-9610-4af464272471" containerID="b7b9e92228bc934febceb25942e7076ab4eec0b9f992787b7dda2d0f213a581b" exitCode=0
Oct 31 03:32:37.346164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:37.345786    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22" event={"ID":"d2ccf889-9140-4bbb-9610-4af464272471","Type":"ContainerDied","Data":"b7b9e92228bc934febceb25942e7076ab4eec0b9f992787b7dda2d0f213a581b"}
Oct 31 03:32:38.673149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.673120    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:38.821020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.820976    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\" (UniqueName: \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\") pod \"d2ccf889-9140-4bbb-9610-4af464272471\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") "
Oct 31 03:32:38.821235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.821220    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\") pod \"d2ccf889-9140-4bbb-9610-4af464272471\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") "
Oct 31 03:32:38.821301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.821284    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\") pod \"d2ccf889-9140-4bbb-9610-4af464272471\" (UID: \"d2ccf889-9140-4bbb-9610-4af464272471\") "
Oct 31 03:32:38.821807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.821768    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca" pod "d2ccf889-9140-4bbb-9610-4af464272471" (UID: "d2ccf889-9140-4bbb-9610-4af464272471"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:38.822468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.822437    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:38.824104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.824071    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf" pod "d2ccf889-9140-4bbb-9610-4af464272471" (UID: "d2ccf889-9140-4bbb-9610-4af464272471"). InnerVolumeSpecName "kube-api-access-cwqqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:38.828075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.828045    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca" pod "d2ccf889-9140-4bbb-9610-4af464272471" (UID: "d2ccf889-9140-4bbb-9610-4af464272471"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:32:38.923876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.923793    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwqqf\" (UniqueName: \"kubernetes.io/projected/d2ccf889-9140-4bbb-9610-4af464272471-kube-api-access-cwqqf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:38.923876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:38.923827    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2ccf889-9140-4bbb-9610-4af464272471-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.330764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.330734    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:32:39.376032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.375996    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"
Oct 31 03:32:39.376032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.376012    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22" event={"ID":"d2ccf889-9140-4bbb-9610-4af464272471","Type":"ContainerDied","Data":"ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e"}
Oct 31 03:32:39.376302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.376070    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd1ad2b3d4b6db7066f9985cb5eab355cf19e7f289c860d36fda338cf2a4f9e"
Oct 31 03:32:39.378618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.378571    5098 generic.go:356] "Generic (PLEG): container finished" podID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerID="8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985" exitCode=0
Oct 31 03:32:39.378777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.378662    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerDied","Data":"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"}
Oct 31 03:32:39.378777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.378677    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"
Oct 31 03:32:39.378777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.378718    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5" event={"ID":"06fae978-b75e-4ef7-97d4-15f857b4afff","Type":"ContainerDied","Data":"0f0b517a31436b7d9b47c78aab0da347bf40be8873dc4e721ca305fe9d91ce6b"}
Oct 31 03:32:39.378777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.378749    5098 scope.go:122] "RemoveContainer" containerID="8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"
Oct 31 03:32:39.399954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.398902    5098 scope.go:122] "RemoveContainer" containerID="5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.421762    5098 scope.go:122] "RemoveContainer" containerID="8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:39.422398    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985\": container with ID starting with 8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985 not found: ID does not exist" containerID="8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.422436    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985"} err="failed to get container status \"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985\": rpc error: code = NotFound desc = could not find container \"8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985\": container with ID starting with 8c6a3c1681fc7d6503e812c07a144d0a4700ca4cda6c423812a1d4da8b9f2985 not found: ID does not exist"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.422464    5098 scope.go:122] "RemoveContainer" containerID="5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:32:39.423276    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116\": container with ID starting with 5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116 not found: ID does not exist" containerID="5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"
Oct 31 03:32:39.439751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.423312    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116"} err="failed to get container status \"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116\": rpc error: code = NotFound desc = could not find container \"5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116\": container with ID starting with 5b3a0cfde0b7e500aa7d8bb459e42382c7c086484180153091c878ce516a5116 not found: ID does not exist"
Oct 31 03:32:39.524852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.524763    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.524896    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.524958    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525051    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525164    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525313    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525393    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525474    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.525543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.525513    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\") pod \"06fae978-b75e-4ef7-97d4-15f857b4afff\" (UID: \"06fae978-b75e-4ef7-97d4-15f857b4afff\") "
Oct 31 03:32:39.526409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.526073    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:39.526409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.526384    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.527295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.526927    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:39.527295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.527120    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:39.527295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.527134    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:39.529947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.529903    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:32:39.531115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.531078    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:39.531301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.531277    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:32:39.532053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.532033    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "kube-api-access-xpbtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:32:39.533856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.533786    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo" pod "06fae978-b75e-4ef7-97d4-15f857b4afff" (UID: "06fae978-b75e-4ef7-97d4-15f857b4afff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:32:39.628661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628588    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06fae978-b75e-4ef7-97d4-15f857b4afff-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628661    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpbtw\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-kube-api-access-xpbtw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628674    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628685    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628694    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628703    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628712    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.628854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.628721    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06fae978-b75e-4ef7-97d4-15f857b4afff-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:32:39.703324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.703270    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:32:39.711355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:39.711319    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-ed9442ec-gw-openshift-default-78db4c97c-j6ts5"]
Oct 31 03:32:41.615731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:41.615688    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" path="/var/lib/kubelet/pods/06fae978-b75e-4ef7-97d4-15f857b4afff/volumes"
Oct 31 03:32:46.291274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:46.291237    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"]
Oct 31 03:32:46.298648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:46.298597    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-72b46d88-qgk22"]
Oct 31 03:32:47.616362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:47.616326    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ccf889-9140-4bbb-9610-4af464272471" path="/var/lib/kubelet/pods/d2ccf889-9140-4bbb-9610-4af464272471/volumes"
Oct 31 03:32:49.087629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.085244    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"]
Oct 31 03:32:49.120637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.119155    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"]
Oct 31 03:32:49.130075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.130024    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6439353e-xxjhh"]
Oct 31 03:32:49.137461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.137426    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81a649c6-86qbq"]
Oct 31 03:32:49.692259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.689485    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c" path="/var/lib/kubelet/pods/4c8254ae-c7d1-4d3c-ac62-b2aa90b2ff9c/volumes"
Oct 31 03:32:49.692259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:49.690313    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e0c5e6-691c-4896-af67-30415a668321" path="/var/lib/kubelet/pods/c2e0c5e6-691c-4896-af67-30415a668321/volumes"
Oct 31 03:32:50.025754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:50.025669    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"]
Oct 31 03:32:50.032544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:50.032512    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dac23050-t6thb"]
Oct 31 03:32:51.664147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:51.664111    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d51d22-8ae5-47f6-bd4c-b165017f91b4" path="/var/lib/kubelet/pods/a8d51d22-8ae5-47f6-bd4c-b165017f91b4/volumes"
Oct 31 03:32:55.752010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:55.751963    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:32:57.082798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:57.082723    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"]
Oct 31 03:32:57.299286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:57.293467    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89d8e9f8-pxpv2"]
Oct 31 03:32:57.933716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:57.861621    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb43f664-561a-4ede-8253-e544b2ee7b0f" path="/var/lib/kubelet/pods/cb43f664-561a-4ede-8253-e544b2ee7b0f/volumes"
Oct 31 03:32:58.301633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:58.280139    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:58.406695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:58.406638    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:58.778241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:58.777885    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:58.900635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:58.899764    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:58.933297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:58.931713    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:59.117316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.117212    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:32:59.401899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.391428    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:32:59.403020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.402976    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:32:59.431633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.405540    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:32:59.515639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.414685    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:32:59.516661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.516627    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:32:59.806638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.804660    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:32:59.952632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.927286    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.003632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:32:59.982741    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:33:00.125698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.117729    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.475694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.422095    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:33:00.526351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.526323    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:33:00.528434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.528410    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:33:00.533762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.533733    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:33:00.657634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.470236    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:33:00.905828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.515766    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.907107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.522987    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.907255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.807014    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:33:00.907344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.907326    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:33:00.907409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.907391    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:33:00.907462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.907434    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:33:00.925698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.818085    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.943630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.882914    5098 patch_prober.go:29] interesting pod/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:33:00.943630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.928125    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:33:00.943630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.939209    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:33:00.950628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.949595    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:33:00.965726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:00.960503    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:33:01.005699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:01.004693    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:33:01.111726 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:33:01.111256    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:33:02.026108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:02.026066    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"]
Oct 31 03:33:02.032800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:02.032762    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-34613fef-4brmm"]
Oct 31 03:33:03.024736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:03.024699    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"]
Oct 31 03:33:03.031034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:03.030999    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79668691-z75hj"]
Oct 31 03:33:03.614037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:03.614003    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac0a529-81e4-4a96-94d7-49a853da4c8b" path="/var/lib/kubelet/pods/1ac0a529-81e4-4a96-94d7-49a853da4c8b/volumes"
Oct 31 03:33:03.614640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:03.614624    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481ab210-f02a-42dc-abba-28979b89e349" path="/var/lib/kubelet/pods/481ab210-f02a-42dc-abba-28979b89e349/volumes"
Oct 31 03:33:05.332782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.332742    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"]
Oct 31 03:33:05.335138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.335103    5098 cpu_manager.go:414] "containerMap: removing container" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.335350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.335330    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.335556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.335539    5098 cpu_manager.go:414] "containerMap: removing container" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.335824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.335663    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.335999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.335984    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d2ccf889-9140-4bbb-9610-4af464272471" containerName="revoke-keys"
Oct 31 03:33:05.336094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.336080    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ccf889-9140-4bbb-9610-4af464272471" containerName="revoke-keys"
Oct 31 03:33:05.336457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.336438    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.336579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.336567    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fae978-b75e-4ef7-97d4-15f857b4afff" containerName="istio-proxy"
Oct 31 03:33:05.336684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.336672    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ccf889-9140-4bbb-9610-4af464272471" containerName="revoke-keys"
Oct 31 03:33:05.344701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.344669    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.354128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.353778    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:33:05.357886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.357149    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"]
Oct 31 03:33:05.359249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.358423    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:33:05.359249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.358552    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:33:05.359249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.358712    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:33:05.470679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.470582    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.471228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.471195    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6pw\" (UniqueName: \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.471363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.471309    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.572917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.572867    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.573122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.573083    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6pw\" (UniqueName: \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.573184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.573155    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.573775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.573742    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.575673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.575644    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.581704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.581664    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6pw\" (UniqueName: \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\") pod \"maas-api-revoke-keys-e2e-ait-0e192091-s8c25\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:05.780171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:05.780134    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:06.025517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:06.025471    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"]
Oct 31 03:33:06.032129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:06.032056    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-541e497c-sn9wh"]
Oct 31 03:33:06.073787 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:33:06.073750    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e92fb7c_1edd_4d75_bf6b_117508f05937.slice/crio-12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6 WatchSource:0}: Error finding container 12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6: Status 404 returned error can't find the container with id 12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6
Oct 31 03:33:06.075818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:06.075787    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"]
Oct 31 03:33:06.433730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:06.433636    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25" event={"ID":"0e92fb7c-1edd-4d75-bf6b-117508f05937","Type":"ContainerStarted","Data":"12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6"}
Oct 31 03:33:07.445873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:07.445768    5098 generic.go:356] "Generic (PLEG): container finished" podID="0e92fb7c-1edd-4d75-bf6b-117508f05937" containerID="edb7b874f974ee2946f2aa18e3f6c01f890bd0f57aefd6c6779b37d5ce29bf4b" exitCode=0
Oct 31 03:33:07.445873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:07.445851    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25" event={"ID":"0e92fb7c-1edd-4d75-bf6b-117508f05937","Type":"ContainerDied","Data":"edb7b874f974ee2946f2aa18e3f6c01f890bd0f57aefd6c6779b37d5ce29bf4b"}
Oct 31 03:33:07.634283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:07.629111    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e633d3fc-b0d4-4b70-83ec-9e578b66772a" path="/var/lib/kubelet/pods/e633d3fc-b0d4-4b70-83ec-9e578b66772a/volumes"
Oct 31 03:33:08.026356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.026314    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"]
Oct 31 03:33:08.032768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.032729    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cf07860-2snv9"]
Oct 31 03:33:08.649350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.649318    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:08.742629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.742539    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\") pod \"0e92fb7c-1edd-4d75-bf6b-117508f05937\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") "
Oct 31 03:33:08.742872 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.742852    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\") pod \"0e92fb7c-1edd-4d75-bf6b-117508f05937\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") "
Oct 31 03:33:08.742968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.742954    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\" (UniqueName: \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\") pod \"0e92fb7c-1edd-4d75-bf6b-117508f05937\" (UID: \"0e92fb7c-1edd-4d75-bf6b-117508f05937\") "
Oct 31 03:33:08.743322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.743253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca" pod "0e92fb7c-1edd-4d75-bf6b-117508f05937" (UID: "0e92fb7c-1edd-4d75-bf6b-117508f05937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:08.743994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.743969    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:08.745006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.744980    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca" pod "0e92fb7c-1edd-4d75-bf6b-117508f05937" (UID: "0e92fb7c-1edd-4d75-bf6b-117508f05937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:08.745748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.745505    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw" pod "0e92fb7c-1edd-4d75-bf6b-117508f05937" (UID: "0e92fb7c-1edd-4d75-bf6b-117508f05937"). InnerVolumeSpecName "kube-api-access-wh6pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:08.845333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.845297    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e92fb7c-1edd-4d75-bf6b-117508f05937-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:08.845333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:08.845329    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6pw\" (UniqueName: \"kubernetes.io/projected/0e92fb7c-1edd-4d75-bf6b-117508f05937-kube-api-access-wh6pw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:09.467677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:09.467638    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"
Oct 31 03:33:09.467677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:09.467654    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25" event={"ID":"0e92fb7c-1edd-4d75-bf6b-117508f05937","Type":"ContainerDied","Data":"12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6"}
Oct 31 03:33:09.467918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:09.467753    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e3d0a9409fafe4297f1b7dc549a1ddadad84701e4a759b75a91821a36063d6"
Oct 31 03:33:09.616936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:09.616894    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ea8b7a-b485-4435-954e-d38b718297b1" path="/var/lib/kubelet/pods/29ea8b7a-b485-4435-954e-d38b718297b1/volumes"
Oct 31 03:33:11.034935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:11.034901    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"]
Oct 31 03:33:11.041158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:11.041125    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a2f9357-874qg"]
Oct 31 03:33:11.615869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:11.615826    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d61abb82-737f-471c-b399-500515b70c1b" path="/var/lib/kubelet/pods/d61abb82-737f-471c-b399-500515b70c1b/volumes"
Oct 31 03:33:17.106819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:17.105667    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"]
Oct 31 03:33:17.116894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:17.116860    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41536206-gq56c"]
Oct 31 03:33:17.613972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:17.613939    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e9e506-4d3f-4b86-ad1d-d62b48e4542f" path="/var/lib/kubelet/pods/75e9e506-4d3f-4b86-ad1d-d62b48e4542f/volumes"
Oct 31 03:33:19.049953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:19.049657    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"]
Oct 31 03:33:19.056104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:19.056074    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e26e762-b6w2c"]
Oct 31 03:33:19.643566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:19.643526    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6df435-a0c8-4d94-94aa-570c4f3e5936" path="/var/lib/kubelet/pods/5e6df435-a0c8-4d94-94aa-570c4f3e5936/volumes"
Oct 31 03:33:20.025498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:20.025414    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"]
Oct 31 03:33:20.042567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:20.042532    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-219891e1-l2vtg"]
Oct 31 03:33:21.029171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.029134    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"]
Oct 31 03:33:21.036426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.036387    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f81ea3e6-j5p6n"]
Oct 31 03:33:21.573748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.573671    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"]
Oct 31 03:33:21.646744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.646704    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4b59b8-d383-459b-87ad-fa8eaf0cbe63" path="/var/lib/kubelet/pods/1c4b59b8-d383-459b-87ad-fa8eaf0cbe63/volumes"
Oct 31 03:33:21.647817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.647791    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b542a9bd-4c75-437f-a939-559b47450e6a" path="/var/lib/kubelet/pods/b542a9bd-4c75-437f-a939-559b47450e6a/volumes"
Oct 31 03:33:21.648529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:21.648510    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0e192091-s8c25"]
Oct 31 03:33:23.613769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:23.613736    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e92fb7c-1edd-4d75-bf6b-117508f05937" path="/var/lib/kubelet/pods/0e92fb7c-1edd-4d75-bf6b-117508f05937/volumes"
Oct 31 03:33:27.868956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.868908    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"]
Oct 31 03:33:27.869705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.869690    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0e92fb7c-1edd-4d75-bf6b-117508f05937" containerName="revoke-keys"
Oct 31 03:33:27.869705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.869705    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e92fb7c-1edd-4d75-bf6b-117508f05937" containerName="revoke-keys"
Oct 31 03:33:27.869839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.869830    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e92fb7c-1edd-4d75-bf6b-117508f05937" containerName="revoke-keys"
Oct 31 03:33:27.875366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.875338    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:27.879334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.879305    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:33:27.880456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.880431    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:33:27.880456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.880431    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:33:27.880677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.880492    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:33:27.881730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:27.881706    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"]
Oct 31 03:33:28.043326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.043279    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.043544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.043352    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.043994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.043864    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk2tc\" (UniqueName: \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.060300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.060264    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"]
Oct 31 03:33:28.068816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.068777    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3cae3c3-qvn9b"]
Oct 31 03:33:28.145101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.145009    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.145101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.145066    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.145484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.145464    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk2tc\" (UniqueName: \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.145925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.145895    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.147835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.147817    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.157067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.157033    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk2tc\" (UniqueName: \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\") pod \"maas-api-revoke-sub-6a50137d-skh2t\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.194267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.194236    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:28.482715 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:33:28.482675    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612204b8_10ee_4033_80ea_e1759e7e13d8.slice/crio-f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710 WatchSource:0}: Error finding container f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710: Status 404 returned error can't find the container with id f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710
Oct 31 03:33:28.487326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.487297    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"]
Oct 31 03:33:28.781464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.781422    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t" event={"ID":"612204b8-10ee-4033-80ea-e1759e7e13d8","Type":"ContainerStarted","Data":"c29157efef871d43bc3bc72b99feb54e2ee9b1732621664ef92f44ce951814a9"}
Oct 31 03:33:28.781730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.781518    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t" event={"ID":"612204b8-10ee-4033-80ea-e1759e7e13d8","Type":"ContainerStarted","Data":"f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710"}
Oct 31 03:33:28.801023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:28.800965    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t" podStartSLOduration=1.800952664 podStartE2EDuration="1.800952664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:33:28.798626966 +0000 UTC m=+1083.323846566" watchObservedRunningTime="2026-10-31 03:33:28.800952664 +0000 UTC m=+1083.326172297"
Oct 31 03:33:29.616463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:29.616426    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="896d7527-15a3-4351-b04d-6d0cc8abfe0f" path="/var/lib/kubelet/pods/896d7527-15a3-4351-b04d-6d0cc8abfe0f/volumes"
Oct 31 03:33:29.822933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:29.822893    5098 generic.go:356] "Generic (PLEG): container finished" podID="612204b8-10ee-4033-80ea-e1759e7e13d8" containerID="c29157efef871d43bc3bc72b99feb54e2ee9b1732621664ef92f44ce951814a9" exitCode=0
Oct 31 03:33:29.823157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:29.823028    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t" event={"ID":"612204b8-10ee-4033-80ea-e1759e7e13d8","Type":"ContainerDied","Data":"c29157efef871d43bc3bc72b99feb54e2ee9b1732621664ef92f44ce951814a9"}
Oct 31 03:33:30.344794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:30.338904    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"]
Oct 31 03:33:30.349627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:30.347511    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b82a4524-4zm57"]
Oct 31 03:33:31.015305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.015278    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:31.030646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.030599    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"]
Oct 31 03:33:31.037700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.037647    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ca791c1-74bl2"]
Oct 31 03:33:31.068508 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:33:31.068469    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:33:31.209682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.205945    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\") pod \"612204b8-10ee-4033-80ea-e1759e7e13d8\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") "
Oct 31 03:33:31.209682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.206074    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\") pod \"612204b8-10ee-4033-80ea-e1759e7e13d8\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") "
Oct 31 03:33:31.209682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.206207    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\" (UniqueName: \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\") pod \"612204b8-10ee-4033-80ea-e1759e7e13d8\" (UID: \"612204b8-10ee-4033-80ea-e1759e7e13d8\") "
Oct 31 03:33:31.233141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.228666    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca" pod "612204b8-10ee-4033-80ea-e1759e7e13d8" (UID: "612204b8-10ee-4033-80ea-e1759e7e13d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:33:31.233141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.230125    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc" pod "612204b8-10ee-4033-80ea-e1759e7e13d8" (UID: "612204b8-10ee-4033-80ea-e1759e7e13d8"). InnerVolumeSpecName "kube-api-access-nk2tc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:33:31.238445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.238410    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca" pod "612204b8-10ee-4033-80ea-e1759e7e13d8" (UID: "612204b8-10ee-4033-80ea-e1759e7e13d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:33:31.309006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.308958    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk2tc\" (UniqueName: \"kubernetes.io/projected/612204b8-10ee-4033-80ea-e1759e7e13d8-kube-api-access-nk2tc\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:31.309006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.308999    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:31.309006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.309011    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/612204b8-10ee-4033-80ea-e1759e7e13d8-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:33:31.624212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.624169    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6" path="/var/lib/kubelet/pods/6ac0fdad-1bd5-4d63-a1e3-61df3a9005f6/volumes"
Oct 31 03:33:31.625012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.624989    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cadd9b3d-46de-4cc6-a180-01adb0b0156b" path="/var/lib/kubelet/pods/cadd9b3d-46de-4cc6-a180-01adb0b0156b/volumes"
Oct 31 03:33:31.878390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.878289    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t" event={"ID":"612204b8-10ee-4033-80ea-e1759e7e13d8","Type":"ContainerDied","Data":"f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710"}
Oct 31 03:33:31.878390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.878363    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f37ea2875be56bd5c7bee583fa339fa626fa269fd2c72480a7da1a759031b710"
Oct 31 03:33:31.878600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:31.878444    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"
Oct 31 03:33:35.027120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:35.027087    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"]
Oct 31 03:33:35.033070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:35.033033    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-720fc28c-xc6cd"]
Oct 31 03:33:35.613913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:35.613871    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b71ba38-3041-4c42-8529-bddde79040e3" path="/var/lib/kubelet/pods/5b71ba38-3041-4c42-8529-bddde79040e3/volumes"
Oct 31 03:33:37.062167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:37.062131    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:33:38.025859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:38.025819    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"]
Oct 31 03:33:38.035983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:38.035938    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9c7f304f-g82dx"]
Oct 31 03:33:39.613677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:39.613595    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a68d99-3008-4468-99de-d799fb336c02" path="/var/lib/kubelet/pods/f6a68d99-3008-4468-99de-d799fb336c02/volumes"
Oct 31 03:33:40.101289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:40.101214    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"]
Oct 31 03:33:40.114993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:40.114958    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-996c90ee-jb6rx"]
Oct 31 03:33:41.614329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:41.614283    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a0e5ba1-4695-48de-9607-3a668b2beff0" path="/var/lib/kubelet/pods/6a0e5ba1-4695-48de-9607-3a668b2beff0/volumes"
Oct 31 03:33:45.086630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:45.079668    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"]
Oct 31 03:33:45.093522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:45.093482    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-692ce641-fd4lp"]
Oct 31 03:33:45.674857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:45.674390    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b835641-630f-4812-9e40-9b0afe0756bb" path="/var/lib/kubelet/pods/0b835641-630f-4812-9e40-9b0afe0756bb/volumes"
Oct 31 03:33:47.024691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:47.024655    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"]
Oct 31 03:33:47.030400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:47.030371    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f4d101f-fsh9k"]
Oct 31 03:33:47.615704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:33:47.615656    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b94fba14-6580-4dab-97b5-b3e2295ee918" path="/var/lib/kubelet/pods/b94fba14-6580-4dab-97b5-b3e2295ee918/volumes"
Oct 31 03:34:01.060790 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:34:01.060728    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:34:09.410777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:09.410669    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:34:10.042736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:10.042693    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"]
Oct 31 03:34:10.057446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:10.057386    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf22eb1a-ngbqn"]
Oct 31 03:34:11.662986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:11.662943    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ab966bf-6818-4368-bf68-04df7c0788f8" path="/var/lib/kubelet/pods/0ab966bf-6818-4368-bf68-04df7c0788f8/volumes"
Oct 31 03:34:24.039078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:24.039003    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"]
Oct 31 03:34:24.040406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:24.040372    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c95da7-kx9cm"]
Oct 31 03:34:25.613908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:25.613873    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7dd4de3-b8e8-425a-a533-7badb0e127d6" path="/var/lib/kubelet/pods/f7dd4de3-b8e8-425a-a533-7badb0e127d6/volumes"
Oct 31 03:34:27.723057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.723010    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:34:27.724623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.724575    5098 cpu_manager.go:414] "containerMap: removing container" podUID="612204b8-10ee-4033-80ea-e1759e7e13d8" containerName="revoke-keys"
Oct 31 03:34:27.724775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.724651    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="612204b8-10ee-4033-80ea-e1759e7e13d8" containerName="revoke-keys"
Oct 31 03:34:27.725158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.725138    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="612204b8-10ee-4033-80ea-e1759e7e13d8" containerName="revoke-keys"
Oct 31 03:34:27.733974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.733939    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.739721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.739686    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50d2d14c-openshift-default-dockercfg-vtnzt\""
Oct 31 03:34:27.746154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.746118    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:34:27.814300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814255    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814309    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814499    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814709    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814833    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814879    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5r9\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.814965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814919    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.815017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.814976    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.815098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.815084    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916568    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5r9\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916632    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916673    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916748    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916831    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.916907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916862    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.916955    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917114    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917295    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917449    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917481    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917431    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.917922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917779    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.918109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.917830    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.919794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.919766    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.919911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.919873    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.925105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.925079    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:27.925319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:27.925297    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5r9\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\") pod \"e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") " pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:28.061985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:28.061942    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:28.389178 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:34:28.389137    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f3475d_f8a1_44fd_9e7b_07a7d609ed11.slice/crio-8b0ef49964cabcd15034eef5199b83f60b78a1e69235682be3b0e5e813db1ca4 WatchSource:0}: Error finding container 8b0ef49964cabcd15034eef5199b83f60b78a1e69235682be3b0e5e813db1ca4: Status 404 returned error can't find the container with id 8b0ef49964cabcd15034eef5199b83f60b78a1e69235682be3b0e5e813db1ca4
Oct 31 03:34:28.390662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:28.390635    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:34:28.392448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:28.392417    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:28.392520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:28.392483    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:28.392520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:28.392510    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:29.146291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:29.146251    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" event={"ID":"72f3475d-f8a1-44fd-9e7b-07a7d609ed11","Type":"ContainerStarted","Data":"a653f74c7b8f48db2ddad825b26bfe0a32323ee746c13f160c4d5afa60b32a65"}
Oct 31 03:34:29.146778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:29.146324    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" event={"ID":"72f3475d-f8a1-44fd-9e7b-07a7d609ed11","Type":"ContainerStarted","Data":"8b0ef49964cabcd15034eef5199b83f60b78a1e69235682be3b0e5e813db1ca4"}
Oct 31 03:34:29.191082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:29.191023    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podStartSLOduration=2.191007304 podStartE2EDuration="2.191007304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:29.188125602 +0000 UTC m=+1143.713345194" watchObservedRunningTime="2026-10-31 03:34:29.191007304 +0000 UTC m=+1143.716226950"
Oct 31 03:34:30.075873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:30.075792    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:30.079052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:30.078641    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:30.079052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:30.078687    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:31.066647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.066098    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:31.066647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.066172    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:31.082224 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:34:31.082179    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:34:31.828639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.823893    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:34:31.839117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.839090    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:31.842866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.842833    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:34:31.843106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:31.842977    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-34cb76-openshift-default-dockercfg-qd8ts\""
Oct 31 03:34:32.021632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.003098    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.069061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.069021    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.069733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.069703    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.069964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.069941    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.070102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.070085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.082681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.070213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.083120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.083085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.083340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.083317    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.083527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.083510    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gl5w\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.148697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.147693    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:32.148697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.147770    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:32.201786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.201745    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.202283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.202248    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.202469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.202449    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.202633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.202596    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.202799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.202781    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.203058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.203041    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.203235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.203221    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.203406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.203384    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gl5w\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.203554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.203534    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.204268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.204244    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.205233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.205211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.205633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.205595    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.224629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.206368    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.229455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.229406    5098 scope.go:122] "RemoveContainer" containerID="5c996a8d3341b9c901156db828b4dc74fa22175bd97cd99e46b4ec74f59c889b"
Oct 31 03:34:32.229940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.209132    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.249371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.236861    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.285085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.285052    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gl5w\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.286511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.286466    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.286897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.286874    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\") pod \"e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") " pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.377143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.377098    5098 scope.go:122] "RemoveContainer" containerID="52c5f917df0258c40fe3fb984d2b6d4b26ae564fc50650964b378f1fa2e53ad5"
Oct 31 03:34:32.424122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.424088    5098 scope.go:122] "RemoveContainer" containerID="93f88c300af62fd5789d161ac218e3629f7328b9d883b7dc552d5593cf15ffff"
Oct 31 03:34:32.436588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.436556    5098 scope.go:122] "RemoveContainer" containerID="b047d347fc5555c4b239435cd10987676ce2620eccb1ffbe9daec4bfb50c62bd"
Oct 31 03:34:32.459887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.459816    5098 scope.go:122] "RemoveContainer" containerID="547b257b696d03b6b200b7c4bc70a950cdf7e09277b4e15c7736bf7186fc5a0e"
Oct 31 03:34:32.493064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.493044    5098 scope.go:122] "RemoveContainer" containerID="32cff6623d858deb2cedc277cc4fe8f5d17b78a25285684c60d56dac57e153a0"
Oct 31 03:34:32.493509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.493391    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:32.521724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.521696    5098 scope.go:122] "RemoveContainer" containerID="42a3e3a384b74a957251b0b99e4a0ca8cd5a187da6a1b08874e2b792d3e6dfc8"
Oct 31 03:34:32.547589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.547566    5098 scope.go:122] "RemoveContainer" containerID="fa96b8f461a326f53398bdf27b1d95a8166731bb70c17c778703f009da5d101d"
Oct 31 03:34:32.595583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.595560    5098 scope.go:122] "RemoveContainer" containerID="6bc8298acb22bb4715b6f4032ff764492626083528e27bedc00cc16a2b8b6493"
Oct 31 03:34:32.612499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.612344    5098 scope.go:122] "RemoveContainer" containerID="8fd67211f53ee8a0b0d2b6f3397ef20a470efee330d53695d6af0ce10e05a585"
Oct 31 03:34:32.703074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.673236    5098 scope.go:122] "RemoveContainer" containerID="8a4fd03dbad5678098671c355d7c435501b5ab879157fed2c28b34ea12e92ad3"
Oct 31 03:34:32.746651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.746575    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"]
Oct 31 03:34:32.770791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.770649    5098 scope.go:122] "RemoveContainer" containerID="85ca7dc02a414e019b512a03335bc91be45144a58bb7902882c67d730a006898"
Oct 31 03:34:32.790882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.790039    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:32.797649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.797061    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:34:32.797649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.797517    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:34:32.797649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.797592    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:34:32.797857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.797658    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"]
Oct 31 03:34:32.801298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.801247    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:34:32.810421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.810380    5098 scope.go:122] "RemoveContainer" containerID="c667487176f0c42ee76e5642675eaff77dc9c0f09de4579106fa38853d72d45b"
Oct 31 03:34:32.838018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.836783    5098 scope.go:122] "RemoveContainer" containerID="962604d8914430c02fc234c3c48b80d195ef36ac8939fffd2b6401a997fa6455"
Oct 31 03:34:32.938091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.938052    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:32.938404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.938374    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chp45\" (UniqueName: \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:32.939008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:32.938981    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.036378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.036203    5098 scope.go:122] "RemoveContainer" containerID="58f476553f54f11ee0e5387b1701f7c8b70e9557112994e534fbc0b12f7a2537"
Oct 31 03:34:33.042261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.042231    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.042415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.042382    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.042496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.042461    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chp45\" (UniqueName: \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.042915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.042894    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.045572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.045534    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.069420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.069375    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:34:33.072922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.071390    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:33.073186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.073157    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:33.081787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.081546    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chp45\" (UniqueName: \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\") pod \"maas-api-revoke-sub-d1892ca4-jdgs4\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.082195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.082048    5098 scope.go:122] "RemoveContainer" containerID="2d25dd4a68debd4891a12aea0184a4e894ddc8337c72fe2c6aeb8809320bd908"
Oct 31 03:34:33.084183 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:34:33.084149    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b27600_cbef_4ca0_9818_265b7b550325.slice/crio-18080a53fe66a519b6c7a23c8a85848d57199b3eee6353ebe71951428db73fb9 WatchSource:0}: Error finding container 18080a53fe66a519b6c7a23c8a85848d57199b3eee6353ebe71951428db73fb9: Status 404 returned error can't find the container with id 18080a53fe66a519b6c7a23c8a85848d57199b3eee6353ebe71951428db73fb9
Oct 31 03:34:33.090644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.090026    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:33.090644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.090181    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:33.090644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.090260    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:34:33.105900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.105860    5098 scope.go:122] "RemoveContainer" containerID="aad0b8af2d60df47b483fbcbe4497cf5debc1ff1f594b444dc689a0f80c3eb6a"
Oct 31 03:34:33.119796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.119765    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:33.420659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.420623    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerStarted","Data":"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"}
Oct 31 03:34:33.420787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.420704    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerStarted","Data":"18080a53fe66a519b6c7a23c8a85848d57199b3eee6353ebe71951428db73fb9"}
Oct 31 03:34:33.453633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.449307    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podStartSLOduration=2.449286016 podStartE2EDuration="2.449286016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:33.443467283 +0000 UTC m=+1147.968686872" watchObservedRunningTime="2026-10-31 03:34:33.449286016 +0000 UTC m=+1147.974505609"
Oct 31 03:34:33.489028 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:34:33.488991    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bff794_a1e0_403b_99cb_d4696b5051d2.slice/crio-30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc WatchSource:0}: Error finding container 30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc: Status 404 returned error can't find the container with id 30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc
Oct 31 03:34:33.494399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.494364    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:33.495147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:33.495113    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"]
Oct 31 03:34:34.070629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.066329    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:34.070629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.066386    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:34.431354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.431237    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4" event={"ID":"06bff794-a1e0-403b-99cb-d4696b5051d2","Type":"ContainerStarted","Data":"727baa2294063c07cbd3b48abaa619d845062e19fb13e95d83e5f7e566646711"}
Oct 31 03:34:34.431354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.431311    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4" event={"ID":"06bff794-a1e0-403b-99cb-d4696b5051d2","Type":"ContainerStarted","Data":"30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc"}
Oct 31 03:34:34.479280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.479213    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4" podStartSLOduration=2.47919289 podStartE2EDuration="2.47919289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:34.447820385 +0000 UTC m=+1148.973039974" watchObservedRunningTime="2026-10-31 03:34:34.47919289 +0000 UTC m=+1149.004412480"
Oct 31 03:34:34.524005 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.523971    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:34.989299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.989259    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"]
Oct 31 03:34:34.994775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:34.994746    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.001690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.001657    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"]
Oct 31 03:34:35.063190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.063153    5098 patch_prober.go:29] interesting pod/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body=
Oct 31 03:34:35.063418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.063213    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused"
Oct 31 03:34:35.192955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.192915    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9f7w\" (UniqueName: \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.192955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.192958    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.193446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.193300    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.306630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.300737    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9f7w\" (UniqueName: \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.306630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.300799    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.306630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.301023    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.306630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.301996    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.327486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.327179    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.335256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.335223    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9f7w\" (UniqueName: \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\") pod \"maas-api-revoke-sub-57285316-pmzjr\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.445542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.445505    5098 generic.go:356] "Generic (PLEG): container finished" podID="06bff794-a1e0-403b-99cb-d4696b5051d2" containerID="727baa2294063c07cbd3b48abaa619d845062e19fb13e95d83e5f7e566646711" exitCode=0
Oct 31 03:34:35.445801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.445673    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4" event={"ID":"06bff794-a1e0-403b-99cb-d4696b5051d2","Type":"ContainerDied","Data":"727baa2294063c07cbd3b48abaa619d845062e19fb13e95d83e5f7e566646711"}
Oct 31 03:34:35.446050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.446023    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:35.447190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.447170    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:34:35.619803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.619711    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:35.921198 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:34:35.921164    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b6cedf_b986_4fdf_b87f_276472caa079.slice/crio-f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a WatchSource:0}: Error finding container f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a: Status 404 returned error can't find the container with id f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a
Oct 31 03:34:35.924172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:35.924144    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"]
Oct 31 03:34:36.068277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.068246    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:36.068803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.068777    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:36.069597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.069574    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:34:36.457759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.457666    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr" event={"ID":"e3b6cedf-b986-4fdf-b87f-276472caa079","Type":"ContainerStarted","Data":"ad6819c150dcdac8f628fcf4c46708b8a400219b56d67a1164891d420f3fd716"}
Oct 31 03:34:36.457759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.457732    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr" event={"ID":"e3b6cedf-b986-4fdf-b87f-276472caa079","Type":"ContainerStarted","Data":"f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a"}
Oct 31 03:34:36.481018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.480942    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr" podStartSLOduration=2.480919661 podStartE2EDuration="2.480919661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:34:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:34:36.474408022 +0000 UTC m=+1150.999627611" watchObservedRunningTime="2026-10-31 03:34:36.480919661 +0000 UTC m=+1151.006139250"
Oct 31 03:34:36.654067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.654039    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:36.786793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.786751    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\") pod \"06bff794-a1e0-403b-99cb-d4696b5051d2\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") "
Oct 31 03:34:36.786985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.786823    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\") pod \"06bff794-a1e0-403b-99cb-d4696b5051d2\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") "
Oct 31 03:34:36.787084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.787072    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\" (UniqueName: \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\") pod \"06bff794-a1e0-403b-99cb-d4696b5051d2\" (UID: \"06bff794-a1e0-403b-99cb-d4696b5051d2\") "
Oct 31 03:34:36.787160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.787129    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca" pod "06bff794-a1e0-403b-99cb-d4696b5051d2" (UID: "06bff794-a1e0-403b-99cb-d4696b5051d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:36.787755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.787726    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:36.789617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.789470    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca" pod "06bff794-a1e0-403b-99cb-d4696b5051d2" (UID: "06bff794-a1e0-403b-99cb-d4696b5051d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:34:36.789819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.789702    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45" pod "06bff794-a1e0-403b-99cb-d4696b5051d2" (UID: "06bff794-a1e0-403b-99cb-d4696b5051d2"). InnerVolumeSpecName "kube-api-access-chp45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:36.921653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.913413    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06bff794-a1e0-403b-99cb-d4696b5051d2-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:36.921653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:36.913451    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chp45\" (UniqueName: \"kubernetes.io/projected/06bff794-a1e0-403b-99cb-d4696b5051d2-kube-api-access-chp45\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:37.701832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:37.701792    5098 generic.go:356] "Generic (PLEG): container finished" podID="e3b6cedf-b986-4fdf-b87f-276472caa079" containerID="ad6819c150dcdac8f628fcf4c46708b8a400219b56d67a1164891d420f3fd716" exitCode=0
Oct 31 03:34:37.705875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:37.705845    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"
Oct 31 03:34:37.708440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:37.708410    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr" event={"ID":"e3b6cedf-b986-4fdf-b87f-276472caa079","Type":"ContainerDied","Data":"ad6819c150dcdac8f628fcf4c46708b8a400219b56d67a1164891d420f3fd716"}
Oct 31 03:34:37.708552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:37.708468    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4" event={"ID":"06bff794-a1e0-403b-99cb-d4696b5051d2","Type":"ContainerDied","Data":"30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc"}
Oct 31 03:34:37.708552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:37.708497    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30314c7faf82a46a5194383691f875f421b9bee8f08d2fc2898abafa4f55a0cc"
Oct 31 03:34:39.004027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.003994    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:39.032381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.032344    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\") pod \"e3b6cedf-b986-4fdf-b87f-276472caa079\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") "
Oct 31 03:34:39.032564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.032423    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\" (UniqueName: \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\") pod \"e3b6cedf-b986-4fdf-b87f-276472caa079\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") "
Oct 31 03:34:39.033079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.032600    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\") pod \"e3b6cedf-b986-4fdf-b87f-276472caa079\" (UID: \"e3b6cedf-b986-4fdf-b87f-276472caa079\") "
Oct 31 03:34:39.033079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.033032    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca" pod "e3b6cedf-b986-4fdf-b87f-276472caa079" (UID: "e3b6cedf-b986-4fdf-b87f-276472caa079"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:34:39.033589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.033541    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:39.035155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.035126    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca" pod "e3b6cedf-b986-4fdf-b87f-276472caa079" (UID: "e3b6cedf-b986-4fdf-b87f-276472caa079"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:34:39.035280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.035198    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w" pod "e3b6cedf-b986-4fdf-b87f-276472caa079" (UID: "e3b6cedf-b986-4fdf-b87f-276472caa079"). InnerVolumeSpecName "kube-api-access-z9f7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:34:39.135432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.135388    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3b6cedf-b986-4fdf-b87f-276472caa079-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:39.135432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.135426    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9f7w\" (UniqueName: \"kubernetes.io/projected/e3b6cedf-b986-4fdf-b87f-276472caa079-kube-api-access-z9f7w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:34:39.723943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.723908    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"
Oct 31 03:34:39.724123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.723947    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr" event={"ID":"e3b6cedf-b986-4fdf-b87f-276472caa079","Type":"ContainerDied","Data":"f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a"}
Oct 31 03:34:39.724123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:39.724001    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f43e404b3a0c18a04dfd238b1afbfaadd8f43b6d2b4855f56b7f54f6637e892a"
Oct 31 03:34:40.086645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:40.079704    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"]
Oct 31 03:34:40.115393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:40.115350    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-684e3d5f-7w5v8"]
Oct 31 03:34:41.613094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:41.613053    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e69120b-d4c3-4d80-b67d-900ea2ab5d73" path="/var/lib/kubelet/pods/9e69120b-d4c3-4d80-b67d-900ea2ab5d73/volumes"
Oct 31 03:34:52.074633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:52.073840    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"]
Oct 31 03:34:52.117631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:52.099570    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a2b2019-phghv"]
Oct 31 03:34:53.614254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:53.614216    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29795917-d8b8-4d77-be74-c20e6ee03f5a" path="/var/lib/kubelet/pods/29795917-d8b8-4d77-be74-c20e6ee03f5a/volumes"
Oct 31 03:34:56.023403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:56.023355    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"]
Oct 31 03:34:56.030156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:56.030119    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0099b050-gjzfz"]
Oct 31 03:34:57.681818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:34:57.681776    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeac5ee5-80ff-4430-aaee-3441dac5062a" path="/var/lib/kubelet/pods/eeac5ee5-80ff-4430-aaee-3441dac5062a/volumes"
Oct 31 03:35:01.028026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:01.027991    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"]
Oct 31 03:35:01.033776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:01.033747    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-826ba39f-6qnxc"]
Oct 31 03:35:01.059279 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:35:01.059240    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:35:01.625740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:01.625700    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1eb1148-4308-44bf-a06f-57bc4d345c42" path="/var/lib/kubelet/pods/f1eb1148-4308-44bf-a06f-57bc4d345c42/volumes"
Oct 31 03:35:05.110982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.110946    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"]
Oct 31 03:35:05.111873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.111852    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e3b6cedf-b986-4fdf-b87f-276472caa079" containerName="revoke-keys"
Oct 31 03:35:05.111873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.111874    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3b6cedf-b986-4fdf-b87f-276472caa079" containerName="revoke-keys"
Oct 31 03:35:05.111979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.111930    5098 cpu_manager.go:414] "containerMap: removing container" podUID="06bff794-a1e0-403b-99cb-d4696b5051d2" containerName="revoke-keys"
Oct 31 03:35:05.111979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.111936    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06bff794-a1e0-403b-99cb-d4696b5051d2" containerName="revoke-keys"
Oct 31 03:35:05.112053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.112044    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3b6cedf-b986-4fdf-b87f-276472caa079" containerName="revoke-keys"
Oct 31 03:35:05.112094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.112060    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="06bff794-a1e0-403b-99cb-d4696b5051d2" containerName="revoke-keys"
Oct 31 03:35:05.117617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.117576    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.121397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.121358    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:35:05.122450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.122422    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"]
Oct 31 03:35:05.122963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.122941    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:35:05.123182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.123164    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:35:05.123241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.123225    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:35:05.278700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.278646    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.278905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.278839    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.278974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.278960    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d44wp\" (UniqueName: \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.380667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.380577    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.380821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.380734    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.380861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.380824    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d44wp\" (UniqueName: \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.381483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.381453    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.383124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.383100    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.389072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.389043    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d44wp\" (UniqueName: \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-px52q\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.434813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.434769    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:05.729990 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:35:05.729953    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63b0c226_81b6_4cfc_91ee_1c261cd49fce.slice/crio-a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894 WatchSource:0}: Error finding container a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894: Status 404 returned error can't find the container with id a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894
Oct 31 03:35:05.732051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:05.732020    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"]
Oct 31 03:35:06.273221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:06.273180    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q" event={"ID":"63b0c226-81b6-4cfc-91ee-1c261cd49fce","Type":"ContainerStarted","Data":"33bdfc42b4563dc842ba78e66fdb118c5a2251f4af77280293a22d643d3fe33e"}
Oct 31 03:35:06.273686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:06.273234    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q" event={"ID":"63b0c226-81b6-4cfc-91ee-1c261cd49fce","Type":"ContainerStarted","Data":"a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894"}
Oct 31 03:35:06.295132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:06.295075    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q" podStartSLOduration=1.295060377 podStartE2EDuration="1.295060377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:06.290139833 +0000 UTC m=+1180.815359427" watchObservedRunningTime="2026-10-31 03:35:06.295060377 +0000 UTC m=+1180.820279965"
Oct 31 03:35:07.412911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:07.412819    5098 generic.go:356] "Generic (PLEG): container finished" podID="63b0c226-81b6-4cfc-91ee-1c261cd49fce" containerID="33bdfc42b4563dc842ba78e66fdb118c5a2251f4af77280293a22d643d3fe33e" exitCode=0
Oct 31 03:35:07.414036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:07.413941    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q" event={"ID":"63b0c226-81b6-4cfc-91ee-1c261cd49fce","Type":"ContainerDied","Data":"33bdfc42b4563dc842ba78e66fdb118c5a2251f4af77280293a22d643d3fe33e"}
Oct 31 03:35:08.739663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.739629    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:08.832065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.832027    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\") pod \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") "
Oct 31 03:35:08.832313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.832229    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\") pod \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") "
Oct 31 03:35:08.832513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.832477    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca" pod "63b0c226-81b6-4cfc-91ee-1c261cd49fce" (UID: "63b0c226-81b6-4cfc-91ee-1c261cd49fce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:08.833430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.833398    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\" (UniqueName: \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\") pod \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\" (UID: \"63b0c226-81b6-4cfc-91ee-1c261cd49fce\") "
Oct 31 03:35:08.834290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.834266    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:08.835683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.835657    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca" pod "63b0c226-81b6-4cfc-91ee-1c261cd49fce" (UID: "63b0c226-81b6-4cfc-91ee-1c261cd49fce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:08.835778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.835740    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp" pod "63b0c226-81b6-4cfc-91ee-1c261cd49fce" (UID: "63b0c226-81b6-4cfc-91ee-1c261cd49fce"). InnerVolumeSpecName "kube-api-access-d44wp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:08.940304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.937384    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63b0c226-81b6-4cfc-91ee-1c261cd49fce-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:08.940304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:08.937430    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d44wp\" (UniqueName: \"kubernetes.io/projected/63b0c226-81b6-4cfc-91ee-1c261cd49fce-kube-api-access-d44wp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:09.527633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:09.525847    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"
Oct 31 03:35:09.527633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:09.525906    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q" event={"ID":"63b0c226-81b6-4cfc-91ee-1c261cd49fce","Type":"ContainerDied","Data":"a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894"}
Oct 31 03:35:09.527633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:09.525964    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0882495c9f2da3950b6d1e3af73d707b7a0410ae15a658a2bc28f079d4c0894"
Oct 31 03:35:12.023333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.023288    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 03:35:12.024399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.024378    5098 cpu_manager.go:414] "containerMap: removing container" podUID="63b0c226-81b6-4cfc-91ee-1c261cd49fce" containerName="revoke-keys"
Oct 31 03:35:12.024453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.024401    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63b0c226-81b6-4cfc-91ee-1c261cd49fce" containerName="revoke-keys"
Oct 31 03:35:12.024575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.024566    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="63b0c226-81b6-4cfc-91ee-1c261cd49fce" containerName="revoke-keys"
Oct 31 03:35:12.028666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.028623    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.036966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.036914    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w5-ab21e4-dockercfg-cfsfw\""
Oct 31 03:35:12.037136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.037081    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w5-ab21e4\""
Oct 31 03:35:12.039395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.039359    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 03:35:12.125387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.125354    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.125387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.125391    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8w5f\" (UniqueName: \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.244106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.244062    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.244382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.244357    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8w5f\" (UniqueName: \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.255640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.246036    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.305783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.305751    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8w5f\" (UniqueName: \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\") pod \"payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.351141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.351104    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:12.876472 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:35:12.876438    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f9979b4_4472_4277_96cc_b5f430090bb2.slice/crio-2844d625d57d7e1b581906f8357f2f74266a9ca23e919c1c2d48417d48627917 WatchSource:0}: Error finding container 2844d625d57d7e1b581906f8357f2f74266a9ca23e919c1c2d48417d48627917: Status 404 returned error can't find the container with id 2844d625d57d7e1b581906f8357f2f74266a9ca23e919c1c2d48417d48627917
Oct 31 03:35:12.877820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:12.877792    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 03:35:13.632357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:13.632278    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" event={"ID":"3f9979b4-4472-4277-96cc-b5f430090bb2","Type":"ContainerStarted","Data":"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"}
Oct 31 03:35:13.632357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:13.632346    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" event={"ID":"3f9979b4-4472-4277-96cc-b5f430090bb2","Type":"ContainerStarted","Data":"2844d625d57d7e1b581906f8357f2f74266a9ca23e919c1c2d48417d48627917"}
Oct 31 03:35:13.633254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:13.632386    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:13.661241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:13.661169    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" podStartSLOduration=2.661148171 podStartE2EDuration="2.661148171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:13.657046835 +0000 UTC m=+1188.182266424" watchObservedRunningTime="2026-10-31 03:35:13.661148171 +0000 UTC m=+1188.186367761"
Oct 31 03:35:14.076576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:14.076541    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"]
Oct 31 03:35:14.152060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:14.148641    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c438cdea-z2wnf"]
Oct 31 03:35:15.485322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.485285    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"]
Oct 31 03:35:15.490789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.490755    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.494829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.494793    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:35:15.495223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.495198    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:35:15.496504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.496473    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:35:15.496636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.496555    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:35:15.501218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.501188    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"]
Oct 31 03:35:15.746693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.746012    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.746693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.746065    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f82fr\" (UniqueName: \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.746693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.746136    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.945295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.933867    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.945295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.933925    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f82fr\" (UniqueName: \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.945295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.934020    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:15.946494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:15.946326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:16.043265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:16.043221    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:16.043482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:16.043326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f82fr\" (UniqueName: \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\") pod \"maas-api-revoke-sub-acc26b59-swvjb\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:16.053772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:16.053738    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7add42f2-c873-4393-aa50-a67df59b0276" path="/var/lib/kubelet/pods/7add42f2-c873-4393-aa50-a67df59b0276/volumes"
Oct 31 03:35:16.113697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:16.113657    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:16.410436 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:35:16.410395    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fed25ec_c903_4c58_b647_43aedfb5109b.slice/crio-bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008 WatchSource:0}: Error finding container bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008: Status 404 returned error can't find the container with id bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008
Oct 31 03:35:16.412758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:16.412722    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"]
Oct 31 03:35:17.065216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:17.065174    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb" event={"ID":"8fed25ec-c903-4c58-b647-43aedfb5109b","Type":"ContainerStarted","Data":"5117566ee5e6290696927c2a8678f63172a0f330e81f06625822173453a093fe"}
Oct 31 03:35:17.065711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:17.065248    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb" event={"ID":"8fed25ec-c903-4c58-b647-43aedfb5109b","Type":"ContainerStarted","Data":"bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008"}
Oct 31 03:35:17.086018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:17.085960    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb" podStartSLOduration=2.085943068 podStartE2EDuration="2.085943068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:17.083648217 +0000 UTC m=+1191.608867811" watchObservedRunningTime="2026-10-31 03:35:17.085943068 +0000 UTC m=+1191.611162657"
Oct 31 03:35:18.076084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:18.076049    5098 generic.go:356] "Generic (PLEG): container finished" podID="8fed25ec-c903-4c58-b647-43aedfb5109b" containerID="5117566ee5e6290696927c2a8678f63172a0f330e81f06625822173453a093fe" exitCode=0
Oct 31 03:35:18.076537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:18.076129    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb" event={"ID":"8fed25ec-c903-4c58-b647-43aedfb5109b","Type":"ContainerDied","Data":"5117566ee5e6290696927c2a8678f63172a0f330e81f06625822173453a093fe"}
Oct 31 03:35:19.278390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.278356    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:19.338796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.338689    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\") pod \"8fed25ec-c903-4c58-b647-43aedfb5109b\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") "
Oct 31 03:35:19.338958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.338943    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\") pod \"8fed25ec-c903-4c58-b647-43aedfb5109b\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") "
Oct 31 03:35:19.339031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.339001    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\" (UniqueName: \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\") pod \"8fed25ec-c903-4c58-b647-43aedfb5109b\" (UID: \"8fed25ec-c903-4c58-b647-43aedfb5109b\") "
Oct 31 03:35:19.339182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.339148    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca" pod "8fed25ec-c903-4c58-b647-43aedfb5109b" (UID: "8fed25ec-c903-4c58-b647-43aedfb5109b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:35:19.339829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.339801    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:19.341198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.341170    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca" pod "8fed25ec-c903-4c58-b647-43aedfb5109b" (UID: "8fed25ec-c903-4c58-b647-43aedfb5109b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:35:19.341862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.341833    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr" pod "8fed25ec-c903-4c58-b647-43aedfb5109b" (UID: "8fed25ec-c903-4c58-b647-43aedfb5109b"). InnerVolumeSpecName "kube-api-access-f82fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:35:19.490633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.483210    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fed25ec-c903-4c58-b647-43aedfb5109b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:19.490633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:19.483253    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f82fr\" (UniqueName: \"kubernetes.io/projected/8fed25ec-c903-4c58-b647-43aedfb5109b-kube-api-access-f82fr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:35:20.106409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:20.103038    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"
Oct 31 03:35:20.106409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:20.103047    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb" event={"ID":"8fed25ec-c903-4c58-b647-43aedfb5109b","Type":"ContainerDied","Data":"bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008"}
Oct 31 03:35:20.106409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:20.104072    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb257ed90736ea99eae70a8f735028f2fa1b9c695b5b6a87bf0518c71a9e3008"
Oct 31 03:35:21.088644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:21.084065    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"]
Oct 31 03:35:21.099903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:21.099770    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4761b44-9vmgt"]
Oct 31 03:35:21.614932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:21.614885    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90e676b-f70d-47fb-8905-e35730f65faf" path="/var/lib/kubelet/pods/e90e676b-f70d-47fb-8905-e35730f65faf/volumes"
Oct 31 03:35:24.645319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:24.645285    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 03:35:25.027449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.027403    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:35:25.481570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.481482    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:35:25.482939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.482911    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8fed25ec-c903-4c58-b647-43aedfb5109b" containerName="revoke-keys"
Oct 31 03:35:25.482939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.482936    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fed25ec-c903-4c58-b647-43aedfb5109b" containerName="revoke-keys"
Oct 31 03:35:25.483258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.483119    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fed25ec-c903-4c58-b647-43aedfb5109b" containerName="revoke-keys"
Oct 31 03:35:25.493478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.489053    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.494663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.494086    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:35:25.538771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.538740    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-d4145b\""
Oct 31 03:35:25.599631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.590997    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d4145b-dockercfg-hxd8j\""
Oct 31 03:35:25.671629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.661354    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.671629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.661409    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfv62\" (UniqueName: \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.762817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.762721    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.763040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.763015    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfv62\" (UniqueName: \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.795843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.795805    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:25.857660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:25.840952    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfv62\" (UniqueName: \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\") pod \"payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:26.114274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:26.114241    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-d4145b-dockercfg-hxd8j\""
Oct 31 03:35:26.121364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:26.121336    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:26.418097 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:35:26.418069    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46903010_6eb1_4d77_b01d_7dd67664d8ac.slice/crio-2546377de1ee72a9d30f458ff9ff3911383feeaad435905cf9f782c55bb22ec4 WatchSource:0}: Error finding container 2546377de1ee72a9d30f458ff9ff3911383feeaad435905cf9f782c55bb22ec4: Status 404 returned error can't find the container with id 2546377de1ee72a9d30f458ff9ff3911383feeaad435905cf9f782c55bb22ec4
Oct 31 03:35:26.420094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:26.420068    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:35:26.470593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:26.470550    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" event={"ID":"46903010-6eb1-4d77-b01d-7dd67664d8ac","Type":"ContainerStarted","Data":"2546377de1ee72a9d30f458ff9ff3911383feeaad435905cf9f782c55bb22ec4"}
Oct 31 03:35:27.828632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:27.825119    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" event={"ID":"46903010-6eb1-4d77-b01d-7dd67664d8ac","Type":"ContainerStarted","Data":"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"}
Oct 31 03:35:27.828632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:27.825629    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:27.850602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:27.850536    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" podStartSLOduration=2.850514669 podStartE2EDuration="2.850514669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:27.848677797 +0000 UTC m=+1202.373897391" watchObservedRunningTime="2026-10-31 03:35:27.850514669 +0000 UTC m=+1202.375734259"
Oct 31 03:35:28.214638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:28.213371    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:35:28.229397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:28.229362    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:35:30.927628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:30.923043    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:35:31.110516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:31.110354    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:35:31.149263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:31.148450    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 31 03:35:31.570716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:31.570675    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"]
Oct 31 03:35:32.015681 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:35:32.005483    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:35:32.052820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:32.031126    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d94a2437-6vnh4"]
Oct 31 03:35:32.884459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:32.884412    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"]
Oct 31 03:35:33.101631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.101171    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-px52q"]
Oct 31 03:35:33.491554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.491521    5098 scope.go:122] "RemoveContainer" containerID="bd7a4c3226757cefc2be9a5100df05fb96cb407cdaf2e650f19e507330d49ce3"
Oct 31 03:35:33.504591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.504562    5098 scope.go:122] "RemoveContainer" containerID="8431f881b7650c0a6f851bad0933024a154d24aa47365a08da83637241f0e1e8"
Oct 31 03:35:33.516204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.516184    5098 scope.go:122] "RemoveContainer" containerID="4cfce9bfa276e3dd173517be9e2a9688a67244ac731cc1ed71fb8930d840e6bd"
Oct 31 03:35:33.528201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.528178    5098 scope.go:122] "RemoveContainer" containerID="ac90a6cec8ab24231fd8ae0280b2578b27404a875ee4bccbb3fc8062083edf20"
Oct 31 03:35:33.538708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.538686    5098 scope.go:122] "RemoveContainer" containerID="3cf3c18a7e98c3d210657262ce93ac520c7f9de646d5530bdc9f81a76d21004b"
Oct 31 03:35:33.576582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.567850    5098 scope.go:122] "RemoveContainer" containerID="cf8f484c6860f301cdb23c7679fdc358b5e42c4f27a54b0c27c539cb0ac24590"
Oct 31 03:35:33.615403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.615376    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="182f0f6f-4678-4264-a943-c4aca911b925" path="/var/lib/kubelet/pods/182f0f6f-4678-4264-a943-c4aca911b925/volumes"
Oct 31 03:35:33.616064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:33.616046    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63b0c226-81b6-4cfc-91ee-1c261cd49fce" path="/var/lib/kubelet/pods/63b0c226-81b6-4cfc-91ee-1c261cd49fce/volumes"
Oct 31 03:35:36.497690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:36.496732    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:35:36.497690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:36.496817    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:35:38.912172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:38.912138    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:35:39.040503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.040468    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"]
Oct 31 03:35:39.046510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.046479    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acfa9078-9zvrl"]
Oct 31 03:35:39.614374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.614339    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead68c66-d9c9-4e86-bbd4-83cde2e7ce43" path="/var/lib/kubelet/pods/ead68c66-d9c9-4e86-bbd4-83cde2e7ce43/volumes"
Oct 31 03:35:39.964324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.964288    5098 generic.go:356] "Generic (PLEG): container finished" podID="42b27600-cbef-4ca0-9818-265b7b550325" containerID="aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a" exitCode=0
Oct 31 03:35:39.964811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.964342    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerDied","Data":"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"}
Oct 31 03:35:39.964911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.964896    5098 scope.go:122] "RemoveContainer" containerID="aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"
Oct 31 03:35:39.966826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.966801    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:39.966907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.966868    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:39.966956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:39.966902    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:41.045697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.045010    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"]
Oct 31 03:35:41.100347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.100303    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerStarted","Data":"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"}
Oct 31 03:35:41.103004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.102967    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2dc3b72-7wrrk"]
Oct 31 03:35:41.517683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.511083    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:35:41.517683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.511222    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:41.517683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.511271    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:41.688628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:41.687630    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00d979b4-6629-4d8e-b499-c373911fa1e4" path="/var/lib/kubelet/pods/00d979b4-6629-4d8e-b499-c373911fa1e4/volumes"
Oct 31 03:35:42.493898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:42.493856    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:42.494388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:42.493925    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:43.500631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:43.500571    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:43.501172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:43.500650    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:44.495225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:44.495185    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:44.495414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:44.495242    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:45.178643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:45.175661    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"]
Oct 31 03:35:45.230639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:45.229677    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71058046-mxgxh"]
Oct 31 03:35:45.500696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:45.499586    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:45.500696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:45.499666    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:45.733632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:45.733047    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4d79bea-3f93-462a-9255-460b4e0a1258" path="/var/lib/kubelet/pods/f4d79bea-3f93-462a-9255-460b4e0a1258/volumes"
Oct 31 03:35:46.037971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:46.035427    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"]
Oct 31 03:35:46.044324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:46.044293    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-404746c7-sljnk"]
Oct 31 03:35:46.494323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:46.494223    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body=
Oct 31 03:35:46.494323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:46.494286    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused"
Oct 31 03:35:47.048241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.048206    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"]
Oct 31 03:35:47.078243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.078200    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3e4ad13-szf9b"]
Oct 31 03:35:47.544753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.544722    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:35:47.545999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.545757    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:35:47.547098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.547057    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:35:47.617318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.617283    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1" path="/var/lib/kubelet/pods/89e62ec8-6ee2-42dc-8c61-1e83cbe4eec1/volumes"
Oct 31 03:35:47.618340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:47.618315    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2d914a1-765f-45fb-8cbb-3ea94f495242" path="/var/lib/kubelet/pods/e2d914a1-765f-45fb-8cbb-3ea94f495242/volumes"
Oct 31 03:35:49.950535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:49.950493    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:35:49.970326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:49.970290    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:35:49.970530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:49.970417    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:49.975439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:49.975410    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-86a834-openshift-default-dockercfg-897pd\""
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.135806    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.135890    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.135924    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.135953    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.135989    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.136113    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.136268    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.136382    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.136677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.136539    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.241776    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.241905    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242038    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242198    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242226    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242256    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242290    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.242836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.242404    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.243442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.243395    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.267634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.264832    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.267634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.265123    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.286661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.286623    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.287844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.287817    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.308637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.296347    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.312641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.311970    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.369689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.368356    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.370497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.370433    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\") pod \"e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") " pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.613764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.613725    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:50.998030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:50.997998    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:35:51.005686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.005649    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:51.005794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.005726    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:51.005794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.005760    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:35:51.337833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.337791    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerStarted","Data":"5e36d47f8fcbc6170d63b1bbc61552c27a1bad8e6fde9a95e5b60cd8f49ffc66"}
Oct 31 03:35:51.338041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.337857    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerStarted","Data":"dfa3925811eb1bfb83fe2738fc0c838d81df1de2fdc14dae3ec3125fc6132927"}
Oct 31 03:35:51.364191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.364133    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podStartSLOduration=2.364116706 podStartE2EDuration="2.364116706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:51.357692815 +0000 UTC m=+1225.882912404" watchObservedRunningTime="2026-10-31 03:35:51.364116706 +0000 UTC m=+1225.889336295"
Oct 31 03:35:51.623184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:51.622725    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:52.621512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:52.621479    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:53.357669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:53.357636    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:53.358879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:53.358853    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:35:55.124503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.124462    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:35:55.130035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.130004    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.144448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.144414    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:35:55.150038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.150005    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-86a834\""
Oct 31 03:35:55.228985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.228950    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-86a834-dockercfg-tvqfk\""
Oct 31 03:35:55.338654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.338165    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.338654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.338424    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns2cp\" (UniqueName: \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.515385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.515287    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns2cp\" (UniqueName: \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.585344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.585301    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.624000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.623965    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.635411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.635341    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns2cp\" (UniqueName: \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\") pod \"payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:55.756229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:55.756183    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:56.034174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.034143    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"]
Oct 31 03:35:56.042138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.042097    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13fbda30-7kl4q"]
Oct 31 03:35:56.185696 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:35:56.185663    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e8985f_f4fe_45bd_a46d_1dacb4865275.slice/crio-adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014 WatchSource:0}: Error finding container adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014: Status 404 returned error can't find the container with id adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014
Oct 31 03:35:56.187557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.187532    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:35:56.607217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.607110    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" event={"ID":"f6e8985f-f4fe-45bd-a46d-1dacb4865275","Type":"ContainerStarted","Data":"97130b945d7adb42185eb5540d5024cbcfca57d29d485e24336d426811db80c4"}
Oct 31 03:35:56.607384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.607250    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:35:56.607384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.607290    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" event={"ID":"f6e8985f-f4fe-45bd-a46d-1dacb4865275","Type":"ContainerStarted","Data":"adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014"}
Oct 31 03:35:56.632582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:56.632526    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" podStartSLOduration=1.63251076 podStartE2EDuration="1.63251076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:35:56.629529213 +0000 UTC m=+1231.154748815" watchObservedRunningTime="2026-10-31 03:35:56.63251076 +0000 UTC m=+1231.157730351"
Oct 31 03:35:57.072634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:57.069432    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"]
Oct 31 03:35:57.236629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:57.234797    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a9208d1-jslb8"]
Oct 31 03:35:57.619714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:57.619677    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e92cfe-71a7-4886-b8c3-bd328ccd9e76" path="/var/lib/kubelet/pods/37e92cfe-71a7-4886-b8c3-bd328ccd9e76/volumes"
Oct 31 03:35:57.620723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:57.620690    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d340b93f-df55-466c-bf1b-98da8d6cf842" path="/var/lib/kubelet/pods/d340b93f-df55-466c-bf1b-98da8d6cf842/volumes"
Oct 31 03:35:59.028709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:59.028676    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"]
Oct 31 03:35:59.035417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:59.035384    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-946be3d5-r8tg5"]
Oct 31 03:35:59.614659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:35:59.614590    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddeaa747-ce05-4b9a-860b-8915ded84ba4" path="/var/lib/kubelet/pods/ddeaa747-ce05-4b9a-860b-8915ded84ba4/volumes"
Oct 31 03:36:00.063754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:00.059685    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"]
Oct 31 03:36:00.098648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:00.089580    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4ff7880-t2gdq"]
Oct 31 03:36:01.060451 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:36:01.060413    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:36:01.629373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:01.629328    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660c337e-6270-4021-a101-1345b352f5a9" path="/var/lib/kubelet/pods/660c337e-6270-4021-a101-1345b352f5a9/volumes"
Oct 31 03:36:06.139804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.139743    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"]
Oct 31 03:36:06.150318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.150267    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b4b2ea-gtf6p"]
Oct 31 03:36:06.417299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.412337    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"]
Oct 31 03:36:06.426489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.426450    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"]
Oct 31 03:36:06.428832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.428800    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.435119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.435077    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:36:06.435542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.435523    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:36:06.437374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.436771    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:36:06.437926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.437909    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:36:06.465098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.465048    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vjhv\" (UniqueName: \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.465501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.465477    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.466378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.466349    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.611713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.610535    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.611713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.610769    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vjhv\" (UniqueName: \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.611713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.610979    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.626632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.614118    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.634631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.622787    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.650409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.650295    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vjhv\" (UniqueName: \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\") pod \"maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:06.757115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:06.757021    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:07.062480 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:36:07.062448    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b1863e_e69c_451a_9e85_738ececc40ca.slice/crio-50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566 WatchSource:0}: Error finding container 50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566: Status 404 returned error can't find the container with id 50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566
Oct 31 03:36:07.063986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:07.063957    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"]
Oct 31 03:36:07.614619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:07.614577    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="585adb43-7243-4f67-aa58-5b37ef216ff9" path="/var/lib/kubelet/pods/585adb43-7243-4f67-aa58-5b37ef216ff9/volumes"
Oct 31 03:36:07.639850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:07.639820    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:36:07.986566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:07.986520    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs" event={"ID":"d2b1863e-e69c-451a-9e85-738ececc40ca","Type":"ContainerStarted","Data":"de274da5932ef9c36fdd98ea374e12f0d3d9e58cc66221b9968774ea4f44360b"}
Oct 31 03:36:07.986797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:07.986583    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs" event={"ID":"d2b1863e-e69c-451a-9e85-738ececc40ca","Type":"ContainerStarted","Data":"50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566"}
Oct 31 03:36:08.009504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:08.009443    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs" podStartSLOduration=2.009425648 podStartE2EDuration="2.009425648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:08.005032145 +0000 UTC m=+1242.530251737" watchObservedRunningTime="2026-10-31 03:36:08.009425648 +0000 UTC m=+1242.534645278"
Oct 31 03:36:08.998830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:08.998792    5098 generic.go:356] "Generic (PLEG): container finished" podID="d2b1863e-e69c-451a-9e85-738ececc40ca" containerID="de274da5932ef9c36fdd98ea374e12f0d3d9e58cc66221b9968774ea4f44360b" exitCode=0
Oct 31 03:36:08.999258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:08.998849    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs" event={"ID":"d2b1863e-e69c-451a-9e85-738ececc40ca","Type":"ContainerDied","Data":"de274da5932ef9c36fdd98ea374e12f0d3d9e58cc66221b9968774ea4f44360b"}
Oct 31 03:36:09.032241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:09.032211    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"]
Oct 31 03:36:09.073274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:09.051807    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-857b876a-dwfwl"]
Oct 31 03:36:09.613550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:09.613508    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d343ef19-7452-46cc-9c9c-5241004fa965" path="/var/lib/kubelet/pods/d343ef19-7452-46cc-9c9c-5241004fa965/volumes"
Oct 31 03:36:10.195849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.195815    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:10.286432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.286397    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\") pod \"d2b1863e-e69c-451a-9e85-738ececc40ca\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") "
Oct 31 03:36:10.286679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.286619    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\") pod \"d2b1863e-e69c-451a-9e85-738ececc40ca\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") "
Oct 31 03:36:10.286749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.286739    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\" (UniqueName: \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\") pod \"d2b1863e-e69c-451a-9e85-738ececc40ca\" (UID: \"d2b1863e-e69c-451a-9e85-738ececc40ca\") "
Oct 31 03:36:10.286952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.286923    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca" pod "d2b1863e-e69c-451a-9e85-738ececc40ca" (UID: "d2b1863e-e69c-451a-9e85-738ececc40ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:10.287369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.287352    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:10.288787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.288762    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca" pod "d2b1863e-e69c-451a-9e85-738ececc40ca" (UID: "d2b1863e-e69c-451a-9e85-738ececc40ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:10.288922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.288892    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv" pod "d2b1863e-e69c-451a-9e85-738ececc40ca" (UID: "d2b1863e-e69c-451a-9e85-738ececc40ca"). InnerVolumeSpecName "kube-api-access-8vjhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:10.414700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.397596    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vjhv\" (UniqueName: \"kubernetes.io/projected/d2b1863e-e69c-451a-9e85-738ececc40ca-kube-api-access-8vjhv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:10.415010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:10.414971    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b1863e-e69c-451a-9e85-738ececc40ca-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:11.020031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:11.019996    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"
Oct 31 03:36:11.020031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:11.020024    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs" event={"ID":"d2b1863e-e69c-451a-9e85-738ececc40ca","Type":"ContainerDied","Data":"50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566"}
Oct 31 03:36:11.020266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:11.020071    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50faf30142c91b51e13e03e98f9bee5bb0e42cf971f6dd0fbfd76d69b2e0c566"
Oct 31 03:36:12.027701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:12.027667    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"]
Oct 31 03:36:12.033994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:12.033963    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4caa91bb-8fmtc"]
Oct 31 03:36:13.634059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:13.634022    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b2a915-1c4c-4d7e-8d46-3bb134a46029" path="/var/lib/kubelet/pods/34b2a915-1c4c-4d7e-8d46-3bb134a46029/volumes"
Oct 31 03:36:15.681843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.681804    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"]
Oct 31 03:36:15.682589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.682574    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d2b1863e-e69c-451a-9e85-738ececc40ca" containerName="revoke-keys"
Oct 31 03:36:15.682710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.682590    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b1863e-e69c-451a-9e85-738ececc40ca" containerName="revoke-keys"
Oct 31 03:36:15.682766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.682759    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b1863e-e69c-451a-9e85-738ececc40ca" containerName="revoke-keys"
Oct 31 03:36:15.687260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.687241    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.693814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.693783    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:36:15.694070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.694035    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:36:15.694236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.693994    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:36:15.694236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.694068    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:36:15.699894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.699869    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"]
Oct 31 03:36:15.764114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.764061    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.764359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.764332    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.764553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.764533    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlf8\" (UniqueName: \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.866661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.866616    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.866844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.866761    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlf8\" (UniqueName: \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.866921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.866904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.867404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.867384    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.869391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.869370    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:15.875141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:15.875111    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlf8\" (UniqueName: \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\") pod \"maas-api-revoke-sub-39805b5d-l5g9x\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:16.009951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:16.009857    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:16.027407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:16.027370    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"]
Oct 31 03:36:16.033798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:16.033764    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17dae9d8-xbpcl"]
Oct 31 03:36:16.312987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:16.312949    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"]
Oct 31 03:36:16.313568 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:36:16.313520    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda738cf03_feb0_459e_af9c_7cc303d6672d.slice/crio-c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd WatchSource:0}: Error finding container c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd: Status 404 returned error can't find the container with id c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd
Oct 31 03:36:17.206690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:17.197312    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x" event={"ID":"a738cf03-feb0-459e-af9c-7cc303d6672d","Type":"ContainerStarted","Data":"488923b302ab3f0ff338af97c0617412137af562af9497fee136458809db1ba6"}
Oct 31 03:36:17.206690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:17.197388    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x" event={"ID":"a738cf03-feb0-459e-af9c-7cc303d6672d","Type":"ContainerStarted","Data":"c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd"}
Oct 31 03:36:17.315112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:17.308661    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x" podStartSLOduration=2.308637691 podStartE2EDuration="2.308637691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:17.235561101 +0000 UTC m=+1251.760780748" watchObservedRunningTime="2026-10-31 03:36:17.308637691 +0000 UTC m=+1251.833857280"
Oct 31 03:36:17.614456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:17.614420    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2531a346-e183-4606-bfa1-c4179de689ee" path="/var/lib/kubelet/pods/2531a346-e183-4606-bfa1-c4179de689ee/volumes"
Oct 31 03:36:18.209818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:18.209774    5098 generic.go:356] "Generic (PLEG): container finished" podID="a738cf03-feb0-459e-af9c-7cc303d6672d" containerID="488923b302ab3f0ff338af97c0617412137af562af9497fee136458809db1ba6" exitCode=0
Oct 31 03:36:18.210243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:18.209854    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x" event={"ID":"a738cf03-feb0-459e-af9c-7cc303d6672d","Type":"ContainerDied","Data":"488923b302ab3f0ff338af97c0617412137af562af9497fee136458809db1ba6"}
Oct 31 03:36:19.412599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.412569    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:19.457052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.457015    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\") pod \"a738cf03-feb0-459e-af9c-7cc303d6672d\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") "
Oct 31 03:36:19.457279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.457228    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\") pod \"a738cf03-feb0-459e-af9c-7cc303d6672d\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") "
Oct 31 03:36:19.457279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.457258    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\" (UniqueName: \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\") pod \"a738cf03-feb0-459e-af9c-7cc303d6672d\" (UID: \"a738cf03-feb0-459e-af9c-7cc303d6672d\") "
Oct 31 03:36:19.457686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.457658    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca" pod "a738cf03-feb0-459e-af9c-7cc303d6672d" (UID: "a738cf03-feb0-459e-af9c-7cc303d6672d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:19.457977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.457946    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:19.459852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.459827    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca" pod "a738cf03-feb0-459e-af9c-7cc303d6672d" (UID: "a738cf03-feb0-459e-af9c-7cc303d6672d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:36:19.459943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.459851    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8" pod "a738cf03-feb0-459e-af9c-7cc303d6672d" (UID: "a738cf03-feb0-459e-af9c-7cc303d6672d"). InnerVolumeSpecName "kube-api-access-vjlf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:19.560219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.560177    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjlf8\" (UniqueName: \"kubernetes.io/projected/a738cf03-feb0-459e-af9c-7cc303d6672d-kube-api-access-vjlf8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:19.560219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:19.560213    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a738cf03-feb0-459e-af9c-7cc303d6672d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:20.302684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:20.294583    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x" event={"ID":"a738cf03-feb0-459e-af9c-7cc303d6672d","Type":"ContainerDied","Data":"c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd"}
Oct 31 03:36:20.302684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:20.298715    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7c0091b81e219bedccf9857240c3a08b084d92591d5496aa60734054c459abd"
Oct 31 03:36:20.302684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:20.298802    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"
Oct 31 03:36:20.715304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:20.715211    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:36:20.715833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:20.715416    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy" containerID="cri-o://280bb8eda60e469b904b1a86f13b7575c5c62d9b56d41083dd4c13f4f0ec1162" gracePeriod=30
Oct 31 03:36:23.804631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:23.799414    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"]
Oct 31 03:36:23.867629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:23.866874    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50d2d14c-rckcs"]
Oct 31 03:36:25.619291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:25.619254    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b1863e-e69c-451a-9e85-738ececc40ca" path="/var/lib/kubelet/pods/d2b1863e-e69c-451a-9e85-738ececc40ca/volumes"
Oct 31 03:36:26.280270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.280227    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:36:26.281523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.281503    5098 cpu_manager.go:414] "containerMap: removing container" podUID="a738cf03-feb0-459e-af9c-7cc303d6672d" containerName="revoke-keys"
Oct 31 03:36:26.281590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.281525    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a738cf03-feb0-459e-af9c-7cc303d6672d" containerName="revoke-keys"
Oct 31 03:36:26.282001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.281973    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="a738cf03-feb0-459e-af9c-7cc303d6672d" containerName="revoke-keys"
Oct 31 03:36:26.288476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.288441    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.289131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.289096    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:36:26.295292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.295270    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-fc226c-openshift-default-dockercfg-lh7zn\""
Oct 31 03:36:26.342952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.342914    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.342983    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343094    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd28k\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343120    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343149    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343423    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343520    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343569    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.343597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.343588    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.370449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.370363    5098 generic.go:356] "Generic (PLEG): container finished" podID="78defb67-9daa-4f22-9580-02319bcae539" containerID="280bb8eda60e469b904b1a86f13b7575c5c62d9b56d41083dd4c13f4f0ec1162" exitCode=0
Oct 31 03:36:26.370449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.370434    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" event={"ID":"78defb67-9daa-4f22-9580-02319bcae539","Type":"ContainerDied","Data":"280bb8eda60e469b904b1a86f13b7575c5c62d9b56d41083dd4c13f4f0ec1162"}
Oct 31 03:36:26.472050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.446319    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.472050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.446472    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.472050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.446530    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.472050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.446556    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.472050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.452631    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.472720    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.472814    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.472923    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd28k\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.472950    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.472999    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.474045    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.475057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.474659    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.504843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.492929    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.504843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.493496    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.504843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.498704    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.510302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.510215    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.515640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.515569    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd28k\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.516156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.516129    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\") pod \"e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") " pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.581316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.581289    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:36:26.608681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.608076    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:26.677191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677158    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677238    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677285    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677401    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677552    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677657    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677705    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.677888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677790    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:36:26.677888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677871    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.678094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.677917    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\") pod \"78defb67-9daa-4f22-9580-02319bcae539\" (UID: \"78defb67-9daa-4f22-9580-02319bcae539\") "
Oct 31 03:36:26.678145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.678087    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:26.678925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.678893    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:26.678925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.678904    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.679094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.678951    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/78defb67-9daa-4f22-9580-02319bcae539-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.679198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.679175    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:26.679251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.679190    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:26.685668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.684889    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:36:26.685794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.685745    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:36:26.687105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.686794    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "kube-api-access-gx2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:26.689328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.688303    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token" pod "78defb67-9daa-4f22-9580-02319bcae539" (UID: "78defb67-9daa-4f22-9580-02319bcae539"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:36:26.780451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780417    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx2n6\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-kube-api-access-gx2n6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780452    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780462    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780472    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780483    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/78defb67-9daa-4f22-9580-02319bcae539-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780491    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/78defb67-9daa-4f22-9580-02319bcae539-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.780683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.780500    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78defb67-9daa-4f22-9580-02319bcae539-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:36:26.906179 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:36:26.906140    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f897d9_1668_4075_b31d_2129a857bd9f.slice/crio-02649c1409cf0b413e6d39a158f443e71684125299979090f6bce07071aedf31 WatchSource:0}: Error finding container 02649c1409cf0b413e6d39a158f443e71684125299979090f6bce07071aedf31: Status 404 returned error can't find the container with id 02649c1409cf0b413e6d39a158f443e71684125299979090f6bce07071aedf31
Oct 31 03:36:26.908358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.908330    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:36:26.909791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.909749    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:26.909895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.909838    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:26.909895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:26.909876    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:27.382417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.382370    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerStarted","Data":"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"}
Oct 31 03:36:27.382631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.382512    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerStarted","Data":"02649c1409cf0b413e6d39a158f443e71684125299979090f6bce07071aedf31"}
Oct 31 03:36:27.384089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.384063    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc" event={"ID":"78defb67-9daa-4f22-9580-02319bcae539","Type":"ContainerDied","Data":"1fb01762274befd28b1ae5db23fd09489732abecb39b3e7b99211f0e6aba68c2"}
Oct 31 03:36:27.384208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.384112    5098 scope.go:122] "RemoveContainer" containerID="280bb8eda60e469b904b1a86f13b7575c5c62d9b56d41083dd4c13f4f0ec1162"
Oct 31 03:36:27.384208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.384066    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"
Oct 31 03:36:27.419311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.419257    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podStartSLOduration=2.419242599 podStartE2EDuration="2.419242599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:27.413586035 +0000 UTC m=+1261.938805624" watchObservedRunningTime="2026-10-31 03:36:27.419242599 +0000 UTC m=+1261.944462187"
Oct 31 03:36:27.437686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.437621    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:36:27.444583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.444551    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-bbtlc"]
Oct 31 03:36:27.647659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.647545    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78defb67-9daa-4f22-9580-02319bcae539" path="/var/lib/kubelet/pods/78defb67-9daa-4f22-9580-02319bcae539/volumes"
Oct 31 03:36:27.648125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:27.648105    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:28.614780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:28.614750    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:29.024977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:29.024884    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"]
Oct 31 03:36:29.029978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:29.029951    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3444c45-86bvd"]
Oct 31 03:36:29.405466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:29.405427    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:29.406572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:29.406545    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:36:29.663593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:29.663355    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16e5fdec-290d-4826-9757-2282b078121c" path="/var/lib/kubelet/pods/16e5fdec-290d-4826-9757-2282b078121c/volumes"
Oct 31 03:36:31.063137 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:36:31.063096    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:36:32.030599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:32.030569    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"]
Oct 31 03:36:32.035398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:32.035333    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3184a815-4dtgx"]
Oct 31 03:36:33.616539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.616502    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2c05363-ee13-4551-8c6f-18c748a06fc2" path="/var/lib/kubelet/pods/b2c05363-ee13-4551-8c6f-18c748a06fc2/volumes"
Oct 31 03:36:33.648355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.648315    5098 scope.go:122] "RemoveContainer" containerID="460d6c6648091ef234cdff8df94d4e8f8aca146837b6161861de6f822b00eec7"
Oct 31 03:36:33.660356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.660291    5098 scope.go:122] "RemoveContainer" containerID="d0ab6b414535b58cb6ef7be4e298afe23ed83ba13a7808ae5354245848ba1e9d"
Oct 31 03:36:33.671505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.671480    5098 scope.go:122] "RemoveContainer" containerID="2d163dc3e1d3a20e3f06ecdf39de492a7470167e339d30f0f9ee402fdcd5473e"
Oct 31 03:36:33.686136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.686112    5098 scope.go:122] "RemoveContainer" containerID="1a6b631278c0a00705a9cb6c727b3eb2cc93d13fe93d071cf23b941302a8f25e"
Oct 31 03:36:33.697759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.697729    5098 scope.go:122] "RemoveContainer" containerID="d0c4ea776e44a0051b5cc0fa53bb95111d569b8faa10d86149d8aa82bcbdb326"
Oct 31 03:36:33.710145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.710119    5098 scope.go:122] "RemoveContainer" containerID="1edf2b3932ef5692e6cf7c17ddf4892a5dac55306c8d1651361b986cb5df0601"
Oct 31 03:36:33.722321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.722292    5098 scope.go:122] "RemoveContainer" containerID="00ce380737c9387f45c7ca4a615c524a5f8a78f8d3c47a190ce4de8af5f8fc1e"
Oct 31 03:36:33.733925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.733901    5098 scope.go:122] "RemoveContainer" containerID="e11c4ae833ee54557e5b7dd4c5d0ea502dccc1e4a566cce4d615f796962bdb82"
Oct 31 03:36:33.745138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.745114    5098 scope.go:122] "RemoveContainer" containerID="d9cc1d08d6a5a30e78b5adbbacf8440959345b70e7f8baa12c24804b7da0bc93"
Oct 31 03:36:33.756310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.756289    5098 scope.go:122] "RemoveContainer" containerID="cf6eeddb79d1afb4931a8ad225e2b99b519c3bf842fc7b2a8c6bda8ba90d81ff"
Oct 31 03:36:33.768123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:33.768102    5098 scope.go:122] "RemoveContainer" containerID="38385a1547cc488e3a7e0393695676df68237da3cd0b21e950041d41924d3c01"
Oct 31 03:36:44.001325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:44.001281    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:36:45.101632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:45.101209    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"]
Oct 31 03:36:45.130631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:45.128068    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e83c26c-wgpxl"]
Oct 31 03:36:45.614207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:45.614172    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060860f5-e3fd-41f7-92d7-86a31236be1d" path="/var/lib/kubelet/pods/060860f5-e3fd-41f7-92d7-86a31236be1d/volumes"
Oct 31 03:36:47.811792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.811733    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:47.812942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.812906    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:36:47.875979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.875908    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:36:47.922384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.921269    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:47.961785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.961728    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:47.969084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:47.968979    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:49.031888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:49.031820    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"]
Oct 31 03:36:49.037246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:49.037208    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-650abfd4-4sz6x"]
Oct 31 03:36:49.613736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:49.613699    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284cf55f-8d82-4ba5-a804-85c66cb0c02c" path="/var/lib/kubelet/pods/284cf55f-8d82-4ba5-a804-85c66cb0c02c/volumes"
Oct 31 03:36:54.399880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:54.399828    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:36:54.400620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:54.400572    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:36:56.079449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.079402    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:36:56.080820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.080792    5098 cpu_manager.go:414] "containerMap: removing container" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy"
Oct 31 03:36:56.081016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.080985    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy"
Oct 31 03:36:56.081314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.081294    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="78defb67-9daa-4f22-9580-02319bcae539" containerName="istio-proxy"
Oct 31 03:36:56.087964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.087936    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.099301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.099272    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-b87769-openshift-default-dockercfg-792qw\""
Oct 31 03:36:56.109569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.109530    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:36:56.185294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185250    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.185465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185409    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.185509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185461    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.185693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185669    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2569\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.185822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185801    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.185975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185953    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.186010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.185994    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.186060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.186046    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.186168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.186157    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418437    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418555    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418583    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418623    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418761    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418858    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418927    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.418963    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.438687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.419063    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2569\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.441235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.439713    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.441235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.440404    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.441235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.440680    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.441235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.440922    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.476259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.476208    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.527878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.523488    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.551115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.550873    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.569099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.569057    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.569988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.569932    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2569\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\") pod \"e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") " pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:56.718905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:56.718813    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:57.021336 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:36:57.021297    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b1a180_c76b_4a9d_861f_243e8ed39e70.slice/crio-c51705f8bfbb2214982c2b63ef5723c295a9423662c034a7b59557e1dc6a9935 WatchSource:0}: Error finding container c51705f8bfbb2214982c2b63ef5723c295a9423662c034a7b59557e1dc6a9935: Status 404 returned error can't find the container with id c51705f8bfbb2214982c2b63ef5723c295a9423662c034a7b59557e1dc6a9935
Oct 31 03:36:57.025016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.024970    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:57.025129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.025068    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:57.025129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.025111    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:57.039221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.039027    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:36:57.647303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.647268    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerStarted","Data":"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"}
Oct 31 03:36:57.650037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.647327    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerStarted","Data":"c51705f8bfbb2214982c2b63ef5723c295a9423662c034a7b59557e1dc6a9935"}
Oct 31 03:36:57.671654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.671552    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podStartSLOduration=1.671535681 podStartE2EDuration="1.671535681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:36:57.671397927 +0000 UTC m=+1292.196617521" watchObservedRunningTime="2026-10-31 03:36:57.671535681 +0000 UTC m=+1292.196755269"
Oct 31 03:36:57.719023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:57.718988    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:58.657679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.657644    5098 generic.go:356] "Generic (PLEG): container finished" podID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerID="5e36d47f8fcbc6170d63b1bbc61552c27a1bad8e6fde9a95e5b60cd8f49ffc66" exitCode=0
Oct 31 03:36:58.658095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.657715    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerDied","Data":"5e36d47f8fcbc6170d63b1bbc61552c27a1bad8e6fde9a95e5b60cd8f49ffc66"}
Oct 31 03:36:58.658308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.658291    5098 scope.go:122] "RemoveContainer" containerID="5e36d47f8fcbc6170d63b1bbc61552c27a1bad8e6fde9a95e5b60cd8f49ffc66"
Oct 31 03:36:58.660211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.660178    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:58.660333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.660271    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:58.660333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.660318    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:36:58.728977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:58.728945    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:59.685777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:59.685740    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerStarted","Data":"2648e7ea608ba401cfaff26ed57758ec1d68246db5c3b65d8e8fbddf78d2ce5f"}
Oct 31 03:36:59.686228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:59.686187    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:36:59.687195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:36:59.687171    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:37:00.625632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:00.623151    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:37:00.625632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:00.623303    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:00.625632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:00.623354    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:01.081217 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:37:01.081177    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:37:01.615455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:01.615419    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:01.615800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:01.615732    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:02.616821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:02.616756    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:02.617326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:02.616834    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:03.027147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:03.027119    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"]
Oct 31 03:37:03.032216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:03.032185    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce35864d-7rrwc"]
Oct 31 03:37:03.615156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:03.615119    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:03.615338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:03.615190    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:03.617521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:03.617492    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e593357b-99cf-410f-bd56-61fcbbd4d177" path="/var/lib/kubelet/pods/e593357b-99cf-410f-bd56-61fcbbd4d177/volumes"
Oct 31 03:37:04.615220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:04.615181    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:04.615415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:04.615256    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:05.637630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:05.629882    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body=
Oct 31 03:37:05.637630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:05.629994    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused"
Oct 31 03:37:05.976769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:05.976666    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:37:05.997243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:05.997192    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:37:05.997511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:05.997482    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.004236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.004202    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-afb8be-openshift-default-dockercfg-gvlck\""
Oct 31 03:37:06.035041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035001    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035081    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035205    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035257    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035326    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035417    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035510    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzsz6\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035544    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.035728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.035634    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.177711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.154118    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.178123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.178089    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.178382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.178357    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.178571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.178553    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.178777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.178743    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.203636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.179036    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.204142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.204104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzsz6\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.204341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.204324    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.204476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.204463    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.231032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.230948    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.311839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.285801    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.311839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.183133    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.311839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.183378    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.311839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.286542    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.344642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.324296    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.496184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.475975    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.496184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.477802    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.523322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.523228    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzsz6\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.623701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.620349    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:06.640926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.640895    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:37:06.669773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.643223    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:37:06.677844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:06.677813    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:37:07.179435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.178754    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:07.179435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.178850    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:07.179435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.178900    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:07.197647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.197536    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:37:07.541152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.541103    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerStarted","Data":"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"}
Oct 31 03:37:07.541323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.541198    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerStarted","Data":"c484468807173230a0e1d10c0adc27bb6489d8eadb73293518610a3d7857a39d"}
Oct 31 03:37:07.571571 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.571506    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podStartSLOduration=2.571489708 podStartE2EDuration="2.571489708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:07.565647971 +0000 UTC m=+1302.090867553" watchObservedRunningTime="2026-10-31 03:37:07.571489708 +0000 UTC m=+1302.096709296"
Oct 31 03:37:07.622917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:07.622876    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:08.628969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:08.628935    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:09.582629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.576773    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"]
Oct 31 03:37:09.665577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.665546    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:09.666008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.665663    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:37:09.666008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.665702    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"]
Oct 31 03:37:09.666187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.666168    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.678526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.678494    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:37:09.678782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.678763    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:37:09.678873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.678848    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:37:09.678942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.678886    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:37:09.760394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.760335    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.760896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.760857    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbtg\" (UniqueName: \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.761054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.760923    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.863182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.863094    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.863348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.863250    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbtg\" (UniqueName: \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.863348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.863295    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.863998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.863974    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.865996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.865969    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.872739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.872710    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbtg\" (UniqueName: \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\") pod \"maas-api-revoke-keys-e2e-mt-8afac225-xsk5z\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:09.983482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:09.983441    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:10.281477 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:37:10.281443    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfa6eb3_39b2_49f8_b917_7f489f649247.slice/crio-63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd WatchSource:0}: Error finding container 63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd: Status 404 returned error can't find the container with id 63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd
Oct 31 03:37:10.282597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:10.282572    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"]
Oct 31 03:37:10.763639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:10.763011    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z" event={"ID":"7cfa6eb3-39b2-49f8-b917-7f489f649247","Type":"ContainerStarted","Data":"63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd"}
Oct 31 03:37:11.024555 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.024466    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"]
Oct 31 03:37:11.029408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.029377    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5ad200b-n27d4"]
Oct 31 03:37:11.353359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.353319    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:37:11.358473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.358450    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.374719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.369334    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-afb8be\""
Oct 31 03:37:11.374719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.371845    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-afb8be-dockercfg-wthjg\""
Oct 31 03:37:11.374980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.374954    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:37:11.420166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.420110    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.420514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.420492    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg726\" (UniqueName: \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.527647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.527072    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg726\" (UniqueName: \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.527647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.527349    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.532640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.528683    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.621698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.618494    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg726\" (UniqueName: \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\") pod \"payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:11.757631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.752744    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4598323-4668-4433-b1c2-516a7d390d59" path="/var/lib/kubelet/pods/a4598323-4668-4433-b1c2-516a7d390d59/volumes"
Oct 31 03:37:11.762632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:11.760987    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:12.038540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:12.035291    5098 generic.go:356] "Generic (PLEG): container finished" podID="7cfa6eb3-39b2-49f8-b917-7f489f649247" containerID="6132d2dfe3f70c7a98bbb48a62ad0f30a873040e4f1392a5f71354880bd54dd3" exitCode=0
Oct 31 03:37:12.038540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:12.037074    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z" event={"ID":"7cfa6eb3-39b2-49f8-b917-7f489f649247","Type":"ContainerDied","Data":"6132d2dfe3f70c7a98bbb48a62ad0f30a873040e4f1392a5f71354880bd54dd3"}
Oct 31 03:37:12.339175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:12.339144    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:37:12.362083 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:37:12.362047    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f633e7a_8a73_4ed1_a75f_aa9725c45d5d.slice/crio-ac8aecdc66d3db3cc092805b65ec09e0fc0ca29b21da691c662844af72c3dbd3 WatchSource:0}: Error finding container ac8aecdc66d3db3cc092805b65ec09e0fc0ca29b21da691c662844af72c3dbd3: Status 404 returned error can't find the container with id ac8aecdc66d3db3cc092805b65ec09e0fc0ca29b21da691c662844af72c3dbd3
Oct 31 03:37:13.107316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.107279    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" event={"ID":"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d","Type":"ContainerStarted","Data":"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"}
Oct 31 03:37:13.108137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.108106    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" event={"ID":"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d","Type":"ContainerStarted","Data":"ac8aecdc66d3db3cc092805b65ec09e0fc0ca29b21da691c662844af72c3dbd3"}
Oct 31 03:37:13.109920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.109762    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:13.160443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.160387    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" podStartSLOduration=2.160370259 podStartE2EDuration="2.160370259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:13.155874949 +0000 UTC m=+1307.681094539" watchObservedRunningTime="2026-10-31 03:37:13.160370259 +0000 UTC m=+1307.685589898"
Oct 31 03:37:13.420888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.416462    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:13.520945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.520904    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\" (UniqueName: \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\") pod \"7cfa6eb3-39b2-49f8-b917-7f489f649247\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") "
Oct 31 03:37:13.521173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.520972    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\") pod \"7cfa6eb3-39b2-49f8-b917-7f489f649247\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") "
Oct 31 03:37:13.521173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.520994    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\") pod \"7cfa6eb3-39b2-49f8-b917-7f489f649247\" (UID: \"7cfa6eb3-39b2-49f8-b917-7f489f649247\") "
Oct 31 03:37:13.521509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.521477    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca" pod "7cfa6eb3-39b2-49f8-b917-7f489f649247" (UID: "7cfa6eb3-39b2-49f8-b917-7f489f649247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:13.522156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.522131    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:13.523257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.523227    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca" pod "7cfa6eb3-39b2-49f8-b917-7f489f649247" (UID: "7cfa6eb3-39b2-49f8-b917-7f489f649247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:37:13.523373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.523349    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg" pod "7cfa6eb3-39b2-49f8-b917-7f489f649247" (UID: "7cfa6eb3-39b2-49f8-b917-7f489f649247"). InnerVolumeSpecName "kube-api-access-dbbtg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:13.624166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.624140    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbtg\" (UniqueName: \"kubernetes.io/projected/7cfa6eb3-39b2-49f8-b917-7f489f649247-kube-api-access-dbbtg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:13.624166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:13.624167    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cfa6eb3-39b2-49f8-b917-7f489f649247-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:14.159643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:14.154719    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"
Oct 31 03:37:14.160393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:14.160141    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z" event={"ID":"7cfa6eb3-39b2-49f8-b917-7f489f649247","Type":"ContainerDied","Data":"63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd"}
Oct 31 03:37:14.160393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:14.160220    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63aacd798b4ac15b30cd17e757b7a24f4f89dafa069070d5934b7878e593c4dd"
Oct 31 03:37:15.065435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:15.065396    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:37:15.065657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:15.065632    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy" containerID="cri-o://a653f74c7b8f48db2ddad825b26bfe0a32323ee746c13f160c4d5afa60b32a65" gracePeriod=30
Oct 31 03:37:20.327630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.325470    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:37:20.327630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.325763    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy" containerID="cri-o://b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9" gracePeriod=30
Oct 31 03:37:20.381685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.379984    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:37:20.381685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.380267    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy" containerID="cri-o://df56cac72d640ecfc26027662f746e1a04acf30af824872172b8d6fe0f618a1b" gracePeriod=30
Oct 31 03:37:20.482378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.482334    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:37:20.482712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.482583    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy" containerID="cri-o://dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a" gracePeriod=30
Oct 31 03:37:20.496505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.491419    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:37:20.496505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.491765    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy" containerID="cri-o://858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd" gracePeriod=30
Oct 31 03:37:20.496505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.496487    5098 generic.go:356] "Generic (PLEG): container finished" podID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerID="a653f74c7b8f48db2ddad825b26bfe0a32323ee746c13f160c4d5afa60b32a65" exitCode=0
Oct 31 03:37:20.496505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.496522    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" event={"ID":"72f3475d-f8a1-44fd-9e7b-07a7d609ed11","Type":"ContainerDied","Data":"a653f74c7b8f48db2ddad825b26bfe0a32323ee746c13f160c4d5afa60b32a65"}
Oct 31 03:37:20.716530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.716495    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:37:20.824734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.824698    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.824942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.824753    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.824942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.824883    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825012    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825100    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825172    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825321    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825392    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825429    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\") pod \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\" (UID: \"72f3475d-f8a1-44fd-9e7b-07a7d609ed11\") "
Oct 31 03:37:20.825526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825486    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:20.825982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.825951    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:20.826404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.826380    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.826500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.826410    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.826558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.826527    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:20.826679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.826596    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:20.826881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.826854    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:20.827713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.827688    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:20.831513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.830562    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "kube-api-access-xl5r9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:20.832748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.831617    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:20.832748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.831916    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token" pod "72f3475d-f8a1-44fd-9e7b-07a7d609ed11" (UID: "72f3475d-f8a1-44fd-9e7b-07a7d609ed11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928475    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928514    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928528    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928540    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928552    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928566    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:20.928963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:20.928581    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5r9\" (UniqueName: \"kubernetes.io/projected/72f3475d-f8a1-44fd-9e7b-07a7d609ed11-kube-api-access-xl5r9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:21.525639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:21.518653    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"
Oct 31 03:37:21.525639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:21.525185    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj" event={"ID":"72f3475d-f8a1-44fd-9e7b-07a7d609ed11","Type":"ContainerDied","Data":"8b0ef49964cabcd15034eef5199b83f60b78a1e69235682be3b0e5e813db1ca4"}
Oct 31 03:37:21.525639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:21.525288    5098 scope.go:122] "RemoveContainer" containerID="a653f74c7b8f48db2ddad825b26bfe0a32323ee746c13f160c4d5afa60b32a65"
Oct 31 03:37:21.631519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:21.626822    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:37:21.631519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:21.629704    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50d2d14c-openshift-default-746b4f8845-fstwj"]
Oct 31 03:37:23.712447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:23.712388    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" path="/var/lib/kubelet/pods/72f3475d-f8a1-44fd-9e7b-07a7d609ed11/volumes"
Oct 31 03:37:25.333131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:25.333096    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:37:25.586741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:25.586671    5098 generic.go:356] "Generic (PLEG): container finished" podID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerID="df56cac72d640ecfc26027662f746e1a04acf30af824872172b8d6fe0f618a1b" exitCode=0
Oct 31 03:37:25.586877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:25.586740    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" event={"ID":"46cd7714-1095-4d02-80d3-fdd4c10724e5","Type":"ContainerDied","Data":"df56cac72d640ecfc26027662f746e1a04acf30af824872172b8d6fe0f618a1b"}
Oct 31 03:37:26.375640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.375408    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:37:26.443645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.441207    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:37:26.492222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492189    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492286    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492318    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492344    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492546 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492374    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492546 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492420    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.492765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492739    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.492850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492828    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:26.492964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492944    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.492964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.492870    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.493351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.493327    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.493423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.493365    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\") pod \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\" (UID: \"673a2afe-eb7d-4de1-b1b6-24c8b2508485\") "
Oct 31 03:37:26.494261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.494235    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.494359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.494265    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.494359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.494279    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.494526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.494501    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.494744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.494721    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.496668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.496593    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "kube-api-access-lngdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.498067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.498036    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.501020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.500959    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:26.501717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.501693    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy" pod "673a2afe-eb7d-4de1-b1b6-24c8b2508485" (UID: "673a2afe-eb7d-4de1-b1b6-24c8b2508485"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.539583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.539551    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:37:26.595663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.595596    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.595803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.595703    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.595803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.595754    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.595917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.595824    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596206    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596259    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596290    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596300    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596379    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.596431    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\") pod \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\" (UID: \"2ec62961-a725-4fd6-9ba6-70f0f7afdfee\") "
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597395    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597422    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597440    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597455    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597471    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673a2afe-eb7d-4de1-b1b6-24c8b2508485-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597485    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.597655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.597500    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lngdt\" (UniqueName: \"kubernetes.io/projected/673a2afe-eb7d-4de1-b1b6-24c8b2508485-kube-api-access-lngdt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.598468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.598221    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "kube-api-access-wkbdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.598529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.598503    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.598778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.598752    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.598778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.598761    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.598955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.598923    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.599036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.599019    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:26.599434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.599411    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:26.600007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.599986    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token" pod "2ec62961-a725-4fd6-9ba6-70f0f7afdfee" (UID: "2ec62961-a725-4fd6-9ba6-70f0f7afdfee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.604352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.604323    5098 generic.go:356] "Generic (PLEG): container finished" podID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerID="858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd" exitCode=0
Oct 31 03:37:26.604531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.604387    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"
Oct 31 03:37:26.604531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.604459    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" event={"ID":"9154c52d-f1a1-45a0-803a-c3384b81c35d","Type":"ContainerDied","Data":"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"}
Oct 31 03:37:26.604531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.604502    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk" event={"ID":"9154c52d-f1a1-45a0-803a-c3384b81c35d","Type":"ContainerDied","Data":"6d15ba4503b213e32bcaf0b8a82c03133bf2cbedf7bcdfc7947fd7932bc472eb"}
Oct 31 03:37:26.604531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.604531    5098 scope.go:122] "RemoveContainer" containerID="858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"
Oct 31 03:37:26.607098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.607069    5098 generic.go:356] "Generic (PLEG): container finished" podID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerID="b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9" exitCode=0
Oct 31 03:37:26.607202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.607111    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"
Oct 31 03:37:26.607202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.607154    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" event={"ID":"673a2afe-eb7d-4de1-b1b6-24c8b2508485","Type":"ContainerDied","Data":"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"}
Oct 31 03:37:26.607335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.607208    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s" event={"ID":"673a2afe-eb7d-4de1-b1b6-24c8b2508485","Type":"ContainerDied","Data":"c09aa343c81436d241d336fab4ff5700575da37a9a83d2611f799e0aa344020e"}
Oct 31 03:37:26.607335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.607279    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:37:26.609543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.609517    5098 generic.go:356] "Generic (PLEG): container finished" podID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerID="dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a" exitCode=0
Oct 31 03:37:26.609543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.609536    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"
Oct 31 03:37:26.609740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.609590    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" event={"ID":"2ec62961-a725-4fd6-9ba6-70f0f7afdfee","Type":"ContainerDied","Data":"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"}
Oct 31 03:37:26.609740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.609705    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd" event={"ID":"2ec62961-a725-4fd6-9ba6-70f0f7afdfee","Type":"ContainerDied","Data":"f1e788cdafa57a8e54e0c7b48f962201f1b63806d477a190e6176279d5e52ab1"}
Oct 31 03:37:26.619326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.619202    5098 scope.go:122] "RemoveContainer" containerID="858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"
Oct 31 03:37:26.619675 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:37:26.619643    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd\": container with ID starting with 858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd not found: ID does not exist" containerID="858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"
Oct 31 03:37:26.619811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.619687    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd"} err="failed to get container status \"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd\": rpc error: code = NotFound desc = could not find container \"858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd\": container with ID starting with 858be5995028d1df4c8cf6e8549ab310fb1f4c2cb2c415cde0342537bd5ac8fd not found: ID does not exist"
Oct 31 03:37:26.619811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.619712    5098 scope.go:122] "RemoveContainer" containerID="b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"
Oct 31 03:37:26.633154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.633130    5098 scope.go:122] "RemoveContainer" containerID="b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"
Oct 31 03:37:26.633545 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:37:26.633501    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9\": container with ID starting with b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9 not found: ID does not exist" containerID="b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"
Oct 31 03:37:26.633735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.633541    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9"} err="failed to get container status \"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9\": rpc error: code = NotFound desc = could not find container \"b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9\": container with ID starting with b9e86118ffee4faaffeb495f5ed8b839a59ca555800ece8071051e6c6b0fabc9 not found: ID does not exist"
Oct 31 03:37:26.633735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.633566    5098 scope.go:122] "RemoveContainer" containerID="dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"
Oct 31 03:37:26.648261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.648230    5098 scope.go:122] "RemoveContainer" containerID="dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"
Oct 31 03:37:26.648694 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:37:26.648662    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a\": container with ID starting with dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a not found: ID does not exist" containerID="dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"
Oct 31 03:37:26.648859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.648695    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a"} err="failed to get container status \"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a\": rpc error: code = NotFound desc = could not find container \"dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a\": container with ID starting with dfdc92e582ef89e64a03cc04869758b739d78ddc1cba18c7be5b731b52abd88a not found: ID does not exist"
Oct 31 03:37:26.656407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.656372    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:37:26.661406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.661376    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-ttd4s"]
Oct 31 03:37:26.672034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.672004    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:37:26.676796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.676767    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-2qqhd"]
Oct 31 03:37:26.698303 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698262    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698322    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698366    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698385    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698416    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698446    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.698749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698493    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.698749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698525    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.698749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698585    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.698749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698737    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.698967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698864    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.698967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698903    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.698973    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.699082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699003    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.699082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699053    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699093    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.699239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699100    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699177    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\") pod \"46cd7714-1095-4d02-80d3-fdd4c10724e5\" (UID: \"46cd7714-1095-4d02-80d3-fdd4c10724e5\") "
Oct 31 03:37:26.699396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699259    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699291    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.699396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699302    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699408    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:26.699533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699486    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.699762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699535    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.699762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.699684    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\") pod \"9154c52d-f1a1-45a0-803a-c3384b81c35d\" (UID: \"9154c52d-f1a1-45a0-803a-c3384b81c35d\") "
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701058    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701087    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701103    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46cd7714-1095-4d02-80d3-fdd4c10724e5-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701120    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701135    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701148    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701468    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701492    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701507    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701522    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701538    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701555    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkbdz\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-kube-api-access-wkbdz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701569    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701587    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ec62961-a725-4fd6-9ba6-70f0f7afdfee-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701621    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701344    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.701864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701493    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:26.702447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701877    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "kube-api-access-7wc84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.702447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.701998    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.702447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702208    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.702447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702335    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:37:26.702674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702433    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.702674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702485    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "kube-api-access-6kfxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.702674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702504    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token" pod "46cd7714-1095-4d02-80d3-fdd4c10724e5" (UID: "46cd7714-1095-4d02-80d3-fdd4c10724e5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:26.702674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.702621    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:26.704414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.704394    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy" pod "9154c52d-f1a1-45a0-803a-c3384b81c35d" (UID: "9154c52d-f1a1-45a0-803a-c3384b81c35d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:37:26.803478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803439    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803472    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803481    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803492    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803501    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803510    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wc84\" (UniqueName: \"kubernetes.io/projected/9154c52d-f1a1-45a0-803a-c3384b81c35d-kube-api-access-7wc84\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803521    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kfxv\" (UniqueName: \"kubernetes.io/projected/46cd7714-1095-4d02-80d3-fdd4c10724e5-kube-api-access-6kfxv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803530    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803538    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803546    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9154c52d-f1a1-45a0-803a-c3384b81c35d-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:26.803889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:26.803554    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9154c52d-f1a1-45a0-803a-c3384b81c35d-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:27.022804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:27.022735    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:37:27.036658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:27.036592    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fs2sk"]
Oct 31 03:37:28.080537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.076620    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"
Oct 31 03:37:28.108940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.108736    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" path="/var/lib/kubelet/pods/2ec62961-a725-4fd6-9ba6-70f0f7afdfee/volumes"
Oct 31 03:37:28.112390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.111967    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" path="/var/lib/kubelet/pods/673a2afe-eb7d-4de1-b1b6-24c8b2508485/volumes"
Oct 31 03:37:28.113639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.113594    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" path="/var/lib/kubelet/pods/9154c52d-f1a1-45a0-803a-c3384b81c35d/volumes"
Oct 31 03:37:28.114330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.114306    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6" event={"ID":"46cd7714-1095-4d02-80d3-fdd4c10724e5","Type":"ContainerDied","Data":"bcbc66d9e5285cebde6254b66aee3463eb98e5de2e7433f6cacedc68363a083c"}
Oct 31 03:37:28.114417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.114345    5098 scope.go:122] "RemoveContainer" containerID="df56cac72d640ecfc26027662f746e1a04acf30af824872172b8d6fe0f618a1b"
Oct 31 03:37:28.130639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.130594    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:37:28.135873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:28.135844    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-gn5d6"]
Oct 31 03:37:29.614029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:29.613995    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" path="/var/lib/kubelet/pods/46cd7714-1095-4d02-80d3-fdd4c10724e5/volumes"
Oct 31 03:37:31.081843 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:37:31.081796    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:37:31.482119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.482033    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"]
Oct 31 03:37:31.488634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.487918    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8afac225-xsk5z"]
Oct 31 03:37:31.586192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.586152    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"]
Oct 31 03:37:31.619729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.619682    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy"
Oct 31 03:37:31.637060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637015    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy"
Oct 31 03:37:31.637379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637354    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7cfa6eb3-39b2-49f8-b917-7f489f649247" containerName="revoke-keys"
Oct 31 03:37:31.637504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637492    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cfa6eb3-39b2-49f8-b917-7f489f649247" containerName="revoke-keys"
Oct 31 03:37:31.637658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637644    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy"
Oct 31 03:37:31.637769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637755    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy"
Oct 31 03:37:31.637866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637854    5098 cpu_manager.go:414] "containerMap: removing container" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy"
Oct 31 03:37:31.637965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.637953    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy"
Oct 31 03:37:31.658771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.658732    5098 cpu_manager.go:414] "containerMap: removing container" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy"
Oct 31 03:37:31.735778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.735679    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy"
Oct 31 03:37:31.736074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.736055    5098 cpu_manager.go:414] "containerMap: removing container" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy"
Oct 31 03:37:31.736176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.736164    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy"
Oct 31 03:37:31.747637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.740940    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="46cd7714-1095-4d02-80d3-fdd4c10724e5" containerName="istio-proxy"
Oct 31 03:37:31.747903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.747876    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ec62961-a725-4fd6-9ba6-70f0f7afdfee" containerName="istio-proxy"
Oct 31 03:37:31.835704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.835665    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cfa6eb3-39b2-49f8-b917-7f489f649247" containerName="revoke-keys"
Oct 31 03:37:31.836319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.836290    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9154c52d-f1a1-45a0-803a-c3384b81c35d" containerName="istio-proxy"
Oct 31 03:37:31.837707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.837686    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f3475d-f8a1-44fd-9e7b-07a7d609ed11" containerName="istio-proxy"
Oct 31 03:37:31.837858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.837846    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="673a2afe-eb7d-4de1-b1b6-24c8b2508485" containerName="istio-proxy"
Oct 31 03:37:31.990405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:31.990320    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.011241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.011207    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:37:32.041126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.028042    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:37:32.045836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.043004    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:37:32.046110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.043101    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:37:32.046214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.045722    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.046369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.046352    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j868v\" (UniqueName: \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.046718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.046699    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.119533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.119497    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cfa6eb3-39b2-49f8-b917-7f489f649247" path="/var/lib/kubelet/pods/7cfa6eb3-39b2-49f8-b917-7f489f649247/volumes"
Oct 31 03:37:32.119972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.119952    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"]
Oct 31 03:37:32.202370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.202328    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.202528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.202488    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.202528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.202520    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j868v\" (UniqueName: \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.203063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.203040    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.205057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.205031    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.211118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.211090    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j868v\" (UniqueName: \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\") pod \"maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.411775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.411738    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:32.747141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:32.747110    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"]
Oct 31 03:37:33.185683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:33.185640    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq" event={"ID":"3852151b-6b32-4aed-b992-0770fb2ef39a","Type":"ContainerStarted","Data":"cfb9a2d5b3c84b8627bee6a7ec24e329a8f4216970ef53c7a4c2b3e8f3003083"}
Oct 31 03:37:33.995983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:33.995938    5098 scope.go:122] "RemoveContainer" containerID="9165142a80acff69010639cd300c8e6b5b95d2f650f826a4fb3f0f037511e7c4"
Oct 31 03:37:34.009878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.009853    5098 scope.go:122] "RemoveContainer" containerID="a6a26bfe33077cdd5a7942c2efdbb3034cc96d5118e90b27ad1664944ee2a0eb"
Oct 31 03:37:34.026322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.026298    5098 scope.go:122] "RemoveContainer" containerID="239bed7a869ff3d61e6726a5b78c25842211bea5890f5db347637bdd7ac8d3ee"
Oct 31 03:37:34.040221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.040196    5098 scope.go:122] "RemoveContainer" containerID="c9fe9ffda4ae827dc3f94d42711bfa839b894e0cc98c3c374622562df4cf9831"
Oct 31 03:37:34.051099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.051073    5098 scope.go:122] "RemoveContainer" containerID="06e8b32ec6a663f474e9b6620eb77a0e1c96aaab9cbf47bf5cc2b9fc9419b552"
Oct 31 03:37:34.064277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.064249    5098 scope.go:122] "RemoveContainer" containerID="530de47fd8de70733d37c07a183ef6b58dbc6e23667348a430345c0694fcba50"
Oct 31 03:37:34.075434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.075410    5098 scope.go:122] "RemoveContainer" containerID="a5beb4710524c2479bd96201d603dedc916024747140156c1794ce00cf77c78e"
Oct 31 03:37:34.087436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.087413    5098 scope.go:122] "RemoveContainer" containerID="ec57428fd99aa90c84b7ee1bccd11b55ff36d74e843954e71eb5d1f4ca170736"
Oct 31 03:37:34.098678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.098649    5098 scope.go:122] "RemoveContainer" containerID="b86432d64c21a4b345b54db79267bd7e77b3c0793e942112eec183180e2b39cb"
Oct 31 03:37:34.110408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.110380    5098 scope.go:122] "RemoveContainer" containerID="f5fa9dab66725d9d73a8921bd1dd2a2907883d285e8d77ea2da0b7cc6cdb8a4f"
Oct 31 03:37:34.122835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.122802    5098 scope.go:122] "RemoveContainer" containerID="873229428f5899ff8a81d059d99efe3471c93b01fed57a67db2a5e9762d9188d"
Oct 31 03:37:34.134320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.134291    5098 scope.go:122] "RemoveContainer" containerID="92075ce215d3c1b371b1efe85b844741b58cf12e1dbe41ceac1712fc6f63af48"
Oct 31 03:37:34.146371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.146345    5098 scope.go:122] "RemoveContainer" containerID="f9540ca1e9635d9e3e25797ccf1d1b3fff0de9c5c3e69d3573a97883e1bf3b0c"
Oct 31 03:37:34.210813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.210781    5098 generic.go:356] "Generic (PLEG): container finished" podID="3852151b-6b32-4aed-b992-0770fb2ef39a" containerID="b0696479ad5bf2cb3855915692a48f211c8e3da257686341591827eb57f7237a" exitCode=0
Oct 31 03:37:34.211214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:34.210944    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq" event={"ID":"3852151b-6b32-4aed-b992-0770fb2ef39a","Type":"ContainerDied","Data":"b0696479ad5bf2cb3855915692a48f211c8e3da257686341591827eb57f7237a"}
Oct 31 03:37:35.457819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.457782    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:35.486945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.486910    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\" (UniqueName: \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\") pod \"3852151b-6b32-4aed-b992-0770fb2ef39a\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") "
Oct 31 03:37:35.487126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.486968    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\") pod \"3852151b-6b32-4aed-b992-0770fb2ef39a\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") "
Oct 31 03:37:35.487393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.487338    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\") pod \"3852151b-6b32-4aed-b992-0770fb2ef39a\" (UID: \"3852151b-6b32-4aed-b992-0770fb2ef39a\") "
Oct 31 03:37:35.490801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.489043    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca" pod "3852151b-6b32-4aed-b992-0770fb2ef39a" (UID: "3852151b-6b32-4aed-b992-0770fb2ef39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:35.491385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.491334    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v" pod "3852151b-6b32-4aed-b992-0770fb2ef39a" (UID: "3852151b-6b32-4aed-b992-0770fb2ef39a"). InnerVolumeSpecName "kube-api-access-j868v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:35.491780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.491673    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca" pod "3852151b-6b32-4aed-b992-0770fb2ef39a" (UID: "3852151b-6b32-4aed-b992-0770fb2ef39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:37:35.589526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.589476    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:35.589526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.589518    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j868v\" (UniqueName: \"kubernetes.io/projected/3852151b-6b32-4aed-b992-0770fb2ef39a-kube-api-access-j868v\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:35.589526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:35.589530    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3852151b-6b32-4aed-b992-0770fb2ef39a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:36.248531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:36.248500    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"
Oct 31 03:37:36.248531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:36.248508    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq" event={"ID":"3852151b-6b32-4aed-b992-0770fb2ef39a","Type":"ContainerDied","Data":"cfb9a2d5b3c84b8627bee6a7ec24e329a8f4216970ef53c7a4c2b3e8f3003083"}
Oct 31 03:37:36.248764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:36.248572    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb9a2d5b3c84b8627bee6a7ec24e329a8f4216970ef53c7a4c2b3e8f3003083"
Oct 31 03:37:37.303173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.303130    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"]
Oct 31 03:37:37.355501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.355461    5098 cpu_manager.go:414] "containerMap: removing container" podUID="3852151b-6b32-4aed-b992-0770fb2ef39a" containerName="revoke-keys"
Oct 31 03:37:37.356405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.356369    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3852151b-6b32-4aed-b992-0770fb2ef39a" containerName="revoke-keys"
Oct 31 03:37:37.356905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.356887    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="3852151b-6b32-4aed-b992-0770fb2ef39a" containerName="revoke-keys"
Oct 31 03:37:37.364020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.363982    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"]
Oct 31 03:37:37.364248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.364117    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.368125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.367945    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:37:37.369528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.369443    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:37:37.369528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.369473    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:37:37.369528 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.369492    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:37:37.473249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.473213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.473493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.473473    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdjxv\" (UniqueName: \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.473600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.473569    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.575557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.575467    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdjxv\" (UniqueName: \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.575753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.575600    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.575826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.575808    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.576317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.576290    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.578326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.578300    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.584346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.584325    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdjxv\" (UniqueName: \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\") pod \"maas-api-revoke-sub-50b44448-rfgpj\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.700182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.700143    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:37.997494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:37.997446    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"]
Oct 31 03:37:38.007113 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:37:38.007071    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4a7a1a_3e83_493b_b8dc_5b0c72bb98ac.slice/crio-5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d WatchSource:0}: Error finding container 5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d: Status 404 returned error can't find the container with id 5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d
Oct 31 03:37:38.347633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:38.342687    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj" event={"ID":"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac","Type":"ContainerStarted","Data":"5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d"}
Oct 31 03:37:39.353693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.353650    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj" event={"ID":"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac","Type":"ContainerStarted","Data":"3ccb0d276adc657a63567b3c149a7ab8b2f22fafad8652fb5c7eaaf6e00decbd"}
Oct 31 03:37:39.373111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.373056    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj" podStartSLOduration=2.373037696 podStartE2EDuration="2.373037696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:39.371046468 +0000 UTC m=+1333.896266057" watchObservedRunningTime="2026-10-31 03:37:39.373037696 +0000 UTC m=+1333.898257284"
Oct 31 03:37:39.897280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.897178    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:37:39.903368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.903339    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:39.913918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.909925    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-adb7b7-openshift-default-dockercfg-chcmz\""
Oct 31 03:37:39.916115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:39.916083    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045274    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045328    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045375    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045510    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045550    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.045640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045583    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.046113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045660    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qnmb\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.046113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045864    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.046113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.045898    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.154966    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qnmb\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155191    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155223    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155351    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155417    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155559    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155622    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.155656    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.156164    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.157273    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.160692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.157936    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.161347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.161000    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.161347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.161244    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.191184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.191082    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.211171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.192316    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qnmb\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.217649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.217373    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.221656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.218631    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\") pod \"e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") " pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.239649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.235546    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:40.397309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.394571    5098 generic.go:356] "Generic (PLEG): container finished" podID="74f897d9-1668-4075-b31d-2129a857bd9f" containerID="0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d" exitCode=0
Oct 31 03:37:40.397309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.394776    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerDied","Data":"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"}
Oct 31 03:37:40.401942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.401379    5098 generic.go:356] "Generic (PLEG): container finished" podID="1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" containerID="3ccb0d276adc657a63567b3c149a7ab8b2f22fafad8652fb5c7eaaf6e00decbd" exitCode=0
Oct 31 03:37:40.401942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.401708    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj" event={"ID":"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac","Type":"ContainerDied","Data":"3ccb0d276adc657a63567b3c149a7ab8b2f22fafad8652fb5c7eaaf6e00decbd"}
Oct 31 03:37:40.403983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.403389    5098 scope.go:122] "RemoveContainer" containerID="0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"
Oct 31 03:37:40.407752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.407633    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.407985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.407939    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.408243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.408135    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.612013 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:37:40.611976    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa540e6_e2f3_4e23_aa12_ae4b2d39b63f.slice/crio-575abf8b69de795b9305a313f67efa0149be8ce6d8e9f0f1fe74f41d43ddb8d2 WatchSource:0}: Error finding container 575abf8b69de795b9305a313f67efa0149be8ce6d8e9f0f1fe74f41d43ddb8d2: Status 404 returned error can't find the container with id 575abf8b69de795b9305a313f67efa0149be8ce6d8e9f0f1fe74f41d43ddb8d2
Oct 31 03:37:40.615334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.615293    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.615454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.615386    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.615454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.615424    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:40.616190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.616057    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:37:40.718887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.718361    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:37:40.724790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.724762    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.742307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.741679    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-555bd6-openshift-default-dockercfg-8nd2m\""
Oct 31 03:37:40.754291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.754250    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:37:40.816701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.816661    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.821935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.821899    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.822289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.822264    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.822445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.822425    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7k2c\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.822591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.822576    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.822754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.822737    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.822945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.822930    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.823183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.823157    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.823287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.823204    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.924862    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925046    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925160    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925189    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7k2c\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925224    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925263    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925354    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925458    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.925777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.925490    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.927566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.926774    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.927566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.927077    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.954792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.951570    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.954792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.951866    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.971031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.968302    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.971031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.970109    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:40.979629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:40.975886    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:41.015656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.011309    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:41.015656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.014294    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7k2c\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\") pod \"e2e-api-a-555bd6-openshift-default-7f57586f99-f995q\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") " pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:41.066340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.066032    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:41.439890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.439852    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:37:41.448559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.448528    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.504037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.503787    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-d04aad-openshift-default-dockercfg-ljdjv\""
Oct 31 03:37:41.517429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.510518    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerStarted","Data":"99be3808da0c557e1c783aafdaa5207d0e046a76dd3d17c92c9e6374076fccbb"}
Oct 31 03:37:41.517429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.510584    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerStarted","Data":"575abf8b69de795b9305a313f67efa0149be8ce6d8e9f0f1fe74f41d43ddb8d2"}
Oct 31 03:37:41.577568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.572769    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:37:41.597632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.583643    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podStartSLOduration=2.5836209329999997 podStartE2EDuration="2.583620933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:41.582207072 +0000 UTC m=+1336.107426661" watchObservedRunningTime="2026-10-31 03:37:41.583620933 +0000 UTC m=+1336.108840516"
Oct 31 03:37:41.622236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.622193    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerStarted","Data":"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"}
Oct 31 03:37:41.629313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.629276    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.629465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.629365    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.629465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.629393    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.629465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.629426    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.630200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.630168    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.630318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.630267    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.630399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.630328    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh65l\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.630399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.630364    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.630514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.630456    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.702229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.702193    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:37:41.712543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.712500    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:41.713207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.713179    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:41.713383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.713363    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:41.735163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735203    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh65l\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735241    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735329    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735644    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735711    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735749    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.735819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735779    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.736015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.735895    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.738984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.736792    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.738984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.736867    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.738984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.737051    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.738984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.738096    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.738984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.738844    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.745966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.745932    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.793968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.793931    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh65l\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.798168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.798142    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.828984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.828948    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\") pod \"e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") " pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:41.895631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:41.895435    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:42.026894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.026852    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:42.141489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.141399    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\") pod \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") "
Oct 31 03:37:42.142172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.142149    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\") pod \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") "
Oct 31 03:37:42.142375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.142357    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\" (UniqueName: \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\") pod \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\" (UID: \"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac\") "
Oct 31 03:37:42.143108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.142049    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca" pod "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" (UID: "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:37:42.148884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.148851    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:42.191210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.191150    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv" pod "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" (UID: "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac"). InnerVolumeSpecName "kube-api-access-sdjxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:37:42.193753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.193712    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca" pod "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" (UID: "1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:37:42.237887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.237686    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:42.254383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.253301    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:42.254383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.253343    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdjxv\" (UniqueName: \"kubernetes.io/projected/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac-kube-api-access-sdjxv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:37:42.410759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.410074    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:42.699750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.699654    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:37:42.709892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.708699    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:42.709892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.708766    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:42.711696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.711663    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerStarted","Data":"f22a7bd9d840ba3334a8934b95c637a033d58ad67d90146517d1d488d97955e2"}
Oct 31 03:37:42.712103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.712062    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerStarted","Data":"fd71b801854a6ac307812b061569e636d1b6297c64f85d22657a60431f63ebd8"}
Oct 31 03:37:42.714127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.714111    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"
Oct 31 03:37:42.718867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.718843    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj" event={"ID":"1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac","Type":"ContainerDied","Data":"5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d"}
Oct 31 03:37:42.718991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.718892    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b5c33a5314c666269586519e4a7d66e17923f65477782f160d908495ff14b0d"
Oct 31 03:37:42.723113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.720271    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:42.726622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.726360    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:37:42.740073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:42.740017    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podStartSLOduration=2.740001702 podStartE2EDuration="2.740001702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:42.736265377 +0000 UTC m=+1337.261484967" watchObservedRunningTime="2026-10-31 03:37:42.740001702 +0000 UTC m=+1337.265221291"
Oct 31 03:37:43.096256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.094824    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:43.100760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.100477    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:37:43.100760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.100536    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:37:43.101598 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:37:43.100789    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce21f832_9004_4bea_aa30_01340b4c84a2.slice/crio-09ae89c33819a4800686d8387535bef8fd2115d8f78ab2a9b66320d21b8d7d49 WatchSource:0}: Error finding container 09ae89c33819a4800686d8387535bef8fd2115d8f78ab2a9b66320d21b8d7d49: Status 404 returned error can't find the container with id 09ae89c33819a4800686d8387535bef8fd2115d8f78ab2a9b66320d21b8d7d49
Oct 31 03:37:43.106709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.106211    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:43.106709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.106293    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:43.106709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.106337    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:37:43.109312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.109279    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:37:43.628445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.628405    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:43.628770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.628737    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:43.724108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.724069    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerStarted","Data":"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"}
Oct 31 03:37:43.724596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.724134    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerStarted","Data":"09ae89c33819a4800686d8387535bef8fd2115d8f78ab2a9b66320d21b8d7d49"}
Oct 31 03:37:43.747636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:43.747558    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podStartSLOduration=2.74754428 podStartE2EDuration="2.74754428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:37:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:37:43.744755017 +0000 UTC m=+1338.269974607" watchObservedRunningTime="2026-10-31 03:37:43.74754428 +0000 UTC m=+1338.272763869"
Oct 31 03:37:44.024695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.024128    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:44.034522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.034490    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:44.090373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.090035    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:44.406056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.406022    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:37:44.620640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.618663    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:44.620640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.618726    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:44.804636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.794922    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:44.804636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.794974    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:44.804636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.796968    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:37:44.804636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:44.797430    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:37:45.610779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:45.610741    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:45.611009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:45.610798    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:46.609230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:46.609196    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:46.609749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:46.609260    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:47.610029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:47.609987    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body=
Oct 31 03:37:47.610589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:47.610564    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused"
Oct 31 03:37:49.611019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:49.610977    5098 patch_prober.go:29] interesting pod/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:37:49.611490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:49.611036    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:37:49.660163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:49.660130    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:37:49.661939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:37:49.661912    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:38:00.520129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.520091    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:38:00.521334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.521300    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" containerName="revoke-keys"
Oct 31 03:38:00.521334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.521331    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" containerName="revoke-keys"
Oct 31 03:38:00.521539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.521526    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" containerName="revoke-keys"
Oct 31 03:38:00.527482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.527456    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.536787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.536757    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-74c760-openshift-default-dockercfg-k8vdv\""
Oct 31 03:38:00.553195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.553160    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:38:00.645144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645101    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645174    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645241    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645304    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645426    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645597    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645714    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645860    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.645983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.645969    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lln2p\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.751962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.751922    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.770016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.769979    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lln2p\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.782885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.782846    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.783191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.783166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.783333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.783316    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.783583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.783563    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.795753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.795709    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.796470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.796447    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.796721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.796699    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.797276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.768461    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.797388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.784472    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.797468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.796239    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.798096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.785534    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.798249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.798180    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.798373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.785302    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.839052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.839016    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.842484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.842452    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lln2p\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.848642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.844173    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\") pod \"e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") " pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:00.869293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:00.868967    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:01.508288 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:01.507808    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:38:01.738864 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:01.738815    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97e84386_1465_4921_85b6_78d9d0edd9ec.slice/crio-9b62b09377c9cbf3333b7b140d19dc09b5c79fdb016bea610ce105f51cfe4a72 WatchSource:0}: Error finding container 9b62b09377c9cbf3333b7b140d19dc09b5c79fdb016bea610ce105f51cfe4a72: Status 404 returned error can't find the container with id 9b62b09377c9cbf3333b7b140d19dc09b5c79fdb016bea610ce105f51cfe4a72
Oct 31 03:38:01.740215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:01.740189    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:38:01.742176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:01.742137    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:01.742315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:01.742232    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:01.742315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:01.742282    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:02.647798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.647762    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerStarted","Data":"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"}
Oct 31 03:38:02.648034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.648019    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerStarted","Data":"9b62b09377c9cbf3333b7b140d19dc09b5c79fdb016bea610ce105f51cfe4a72"}
Oct 31 03:38:02.702560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.702469    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podStartSLOduration=2.702454244 podStartE2EDuration="2.702454244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:02.700329776 +0000 UTC m=+1357.225549366" watchObservedRunningTime="2026-10-31 03:38:02.702454244 +0000 UTC m=+1357.227673830"
Oct 31 03:38:02.865048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.864992    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:38:02.870696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.870668    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:02.870850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.870758    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:02.882173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.882134    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-34dbeb\""
Oct 31 03:38:02.882752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.882661    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:38:02.888366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.888341    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:38:02.893434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.893410    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:02.902108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.902011    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-34dbeb-dockercfg-zrz9p\""
Oct 31 03:38:02.911130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.911100    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:38:02.968833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.968800    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2444\" (UniqueName: \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:02.969020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.969008    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4txz\" (UniqueName: \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:02.969077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.969041    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:02.969077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:02.969058    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.078633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.072294    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4txz\" (UniqueName: \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:03.078633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.072402    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:03.078633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.072434    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.078633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.072802    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2444\" (UniqueName: \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.078633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.074900    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:03.088647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.079809    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.135634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.134849    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4txz\" (UniqueName: \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\") pod \"payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") " pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:03.160676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.158877    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2444\" (UniqueName: \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\") pod \"payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.202460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.201191    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:03.267051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.267007    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:03.787444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.787397    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:38:03.815032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.815000    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:38:03.825290 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:03.825243    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf2aa246_0acb_4379_b3f4_4f091c4db938.slice/crio-2b42973e8372b6f0be28175a09a6a7968c9d8a80222c9efb31fe3ff8f3639c9b WatchSource:0}: Error finding container 2b42973e8372b6f0be28175a09a6a7968c9d8a80222c9efb31fe3ff8f3639c9b: Status 404 returned error can't find the container with id 2b42973e8372b6f0be28175a09a6a7968c9d8a80222c9efb31fe3ff8f3639c9b
Oct 31 03:38:03.876511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:03.876478    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:04.706385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.706347    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" event={"ID":"bf2aa246-0acb-4379-b3f4-4f091c4db938","Type":"ContainerStarted","Data":"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"}
Oct 31 03:38:04.706648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.706413    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" event={"ID":"bf2aa246-0acb-4379-b3f4-4f091c4db938","Type":"ContainerStarted","Data":"2b42973e8372b6f0be28175a09a6a7968c9d8a80222c9efb31fe3ff8f3639c9b"}
Oct 31 03:38:04.706648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.706443    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:04.708468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.708433    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" event={"ID":"39710305-ec38-4c9a-89e4-e8d0b8652414","Type":"ContainerStarted","Data":"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"}
Oct 31 03:38:04.708570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.708493    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" event={"ID":"39710305-ec38-4c9a-89e4-e8d0b8652414","Type":"ContainerStarted","Data":"895597c915193c92c53fbdc3ed35b3427f239459dafb6ef22bcf0d70e5d16fe7"}
Oct 31 03:38:04.708680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.708662    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:04.708731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.708720    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:04.709569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.709551    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:38:04.725416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.725365    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" podStartSLOduration=2.725351229 podStartE2EDuration="2.725351229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:04.723148135 +0000 UTC m=+1359.248367738" watchObservedRunningTime="2026-10-31 03:38:04.725351229 +0000 UTC m=+1359.250570867"
Oct 31 03:38:04.761785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:04.761725    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" podStartSLOduration=2.761708502 podStartE2EDuration="2.761708502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:04.76102275 +0000 UTC m=+1359.286242354" watchObservedRunningTime="2026-10-31 03:38:04.761708502 +0000 UTC m=+1359.286928135"
Oct 31 03:38:07.254450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.254406    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"]
Oct 31 03:38:07.688361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.688319    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fc226c-xqjmq"]
Oct 31 03:38:07.688577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.688379    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:38:07.697842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.697803    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:38:07.698021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.697913    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.702132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.701985    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-74c760\""
Oct 31 03:38:07.744773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.744743    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-74c760-dockercfg-rzbtp\""
Oct 31 03:38:07.823202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.823161    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.823597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.823558    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwq6m\" (UniqueName: \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.888833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.888799    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:38:07.926080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.926046    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.926269 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.926217    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwq6m\" (UniqueName: \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.926782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.926762    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:07.935227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:07.935199    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwq6m\" (UniqueName: \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\") pod \"payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:08.019250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.017934    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:08.638850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.638171    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:38:08.961679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.960879    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" event={"ID":"ecc0acca-8037-4b84-9263-b32a193fd85a","Type":"ContainerStarted","Data":"f0fbbd0559787b5fb5315b3ea81609181f6df44fce5a808ed3ba810f91a048e6"}
Oct 31 03:38:08.961679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.961070    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:08.961679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.961099    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" event={"ID":"ecc0acca-8037-4b84-9263-b32a193fd85a","Type":"ContainerStarted","Data":"5d5be4f1e23edec079ffccd4296035b2236d6bb9404f6dfa96570fc7751d8aaa"}
Oct 31 03:38:08.981935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:08.981881    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" podStartSLOduration=1.9818662040000001 podStartE2EDuration="1.981866204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:08.978931849 +0000 UTC m=+1363.504151437" watchObservedRunningTime="2026-10-31 03:38:08.981866204 +0000 UTC m=+1363.507085793"
Oct 31 03:38:09.670018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:09.669981    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3852151b-6b32-4aed-b992-0770fb2ef39a" path="/var/lib/kubelet/pods/3852151b-6b32-4aed-b992-0770fb2ef39a/volumes"
Oct 31 03:38:13.820620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:13.820568    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:38:13.823434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:13.823361    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy" containerID="cri-o://f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624" gracePeriod=30
Oct 31 03:38:14.361667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.355906    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"]
Oct 31 03:38:14.396636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.382313    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.402708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.401453    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:38:14.402708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.401788    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:38:14.404973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.403195    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:38:14.404973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.403315    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:38:14.408660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.408592    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"]
Oct 31 03:38:14.500646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.497310    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.500646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.497467    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvgsv\" (UniqueName: \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.524637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.514181    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.622679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.620456    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.622679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.620583    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.622679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.620707    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvgsv\" (UniqueName: \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.623628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.623460    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.624162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.624136    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.630658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.630596    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvgsv\" (UniqueName: \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\") pod \"maas-api-revoke-keys-e2e-api-b-b87769-sqq7r\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:14.721574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:14.721536    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:15.040184 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:15.040154    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71aa485d_c994_4af0_935b_ca22e1616723.slice/crio-b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25 WatchSource:0}: Error finding container b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25: Status 404 returned error can't find the container with id b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25
Oct 31 03:38:15.040813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:15.040785    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"]
Oct 31 03:38:15.238951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:15.238919    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r" event={"ID":"71aa485d-c994-4af0-935b-ca22e1616723","Type":"ContainerStarted","Data":"b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25"}
Oct 31 03:38:15.720274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:15.720237    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:38:15.720459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:15.720318    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:38:16.250467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:16.250430    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r" event={"ID":"71aa485d-c994-4af0-935b-ca22e1616723","Type":"ContainerStarted","Data":"e08d461462a97aaac66e2e4e66d3d23acde399e4696a782d167ac461a85725cf"}
Oct 31 03:38:16.269890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:16.269822    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r" podStartSLOduration=2.269801773 podStartE2EDuration="2.269801773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:16.268930294 +0000 UTC m=+1370.794149882" watchObservedRunningTime="2026-10-31 03:38:16.269801773 +0000 UTC m=+1370.795021362"
Oct 31 03:38:17.263255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:17.263223    5098 generic.go:356] "Generic (PLEG): container finished" podID="71aa485d-c994-4af0-935b-ca22e1616723" containerID="e08d461462a97aaac66e2e4e66d3d23acde399e4696a782d167ac461a85725cf" exitCode=0
Oct 31 03:38:17.263686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:17.263302    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r" event={"ID":"71aa485d-c994-4af0-935b-ca22e1616723","Type":"ContainerDied","Data":"e08d461462a97aaac66e2e4e66d3d23acde399e4696a782d167ac461a85725cf"}
Oct 31 03:38:18.981342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:18.981038    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:19.066242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.060206    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\") pod \"71aa485d-c994-4af0-935b-ca22e1616723\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") "
Oct 31 03:38:19.066242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.060320    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\" (UniqueName: \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\") pod \"71aa485d-c994-4af0-935b-ca22e1616723\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") "
Oct 31 03:38:19.066242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.060818    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\") pod \"71aa485d-c994-4af0-935b-ca22e1616723\" (UID: \"71aa485d-c994-4af0-935b-ca22e1616723\") "
Oct 31 03:38:19.066242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.062180    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca" pod "71aa485d-c994-4af0-935b-ca22e1616723" (UID: "71aa485d-c994-4af0-935b-ca22e1616723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:19.079678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.078079    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca" pod "71aa485d-c994-4af0-935b-ca22e1616723" (UID: "71aa485d-c994-4af0-935b-ca22e1616723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:38:19.080384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.080348    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv" pod "71aa485d-c994-4af0-935b-ca22e1616723" (UID: "71aa485d-c994-4af0-935b-ca22e1616723"). InnerVolumeSpecName "kube-api-access-pvgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.163355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.163325    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.163355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.163353    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71aa485d-c994-4af0-935b-ca22e1616723-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.163355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.163363    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvgsv\" (UniqueName: \"kubernetes.io/projected/71aa485d-c994-4af0-935b-ca22e1616723-kube-api-access-pvgsv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.166939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.166914    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:38:19.265670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.265509    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266443    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266508    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266563    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266633    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266671    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.266859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.266852    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.267071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.267042    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.267146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.267126    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\") pod \"74f897d9-1668-4075-b31d-2129a857bd9f\" (UID: \"74f897d9-1668-4075-b31d-2129a857bd9f\") "
Oct 31 03:38:19.268651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.268617    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.268651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.268636    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.268867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.268829    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.268867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.268850    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:19.269017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.268539    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.269796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.269772    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.270502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.270487    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.270682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.270664    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74f897d9-1668-4075-b31d-2129a857bd9f-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.270832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.270818    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.270946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.270933    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.271176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.270373    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:19.272784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.272744    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:19.276663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.274785    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "kube-api-access-sd28k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.276663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.276141    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token" pod "74f897d9-1668-4075-b31d-2129a857bd9f" (UID: "74f897d9-1668-4075-b31d-2129a857bd9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:19.346759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.346677    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"
Oct 31 03:38:19.346759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.346737    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r" event={"ID":"71aa485d-c994-4af0-935b-ca22e1616723","Type":"ContainerDied","Data":"b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25"}
Oct 31 03:38:19.346997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.346794    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a77884347c8c6a312d14a65831ee253b04ef8ba743caa70fd9d73b811b3f25"
Oct 31 03:38:19.349347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.349310    5098 generic.go:356] "Generic (PLEG): container finished" podID="74f897d9-1668-4075-b31d-2129a857bd9f" containerID="f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624" exitCode=0
Oct 31 03:38:19.349536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.349388    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"
Oct 31 03:38:19.349536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.349391    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerDied","Data":"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"}
Oct 31 03:38:19.349536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.349497    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd" event={"ID":"74f897d9-1668-4075-b31d-2129a857bd9f","Type":"ContainerDied","Data":"02649c1409cf0b413e6d39a158f443e71684125299979090f6bce07071aedf31"}
Oct 31 03:38:19.349536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.349533    5098 scope.go:122] "RemoveContainer" containerID="f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"
Oct 31 03:38:19.368916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.368892    5098 scope.go:122] "RemoveContainer" containerID="0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"
Oct 31 03:38:19.373566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.373538    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74f897d9-1668-4075-b31d-2129a857bd9f-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.373566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.373568    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.373737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.373581    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74f897d9-1668-4075-b31d-2129a857bd9f-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.373737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.373595    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd28k\" (UniqueName: \"kubernetes.io/projected/74f897d9-1668-4075-b31d-2129a857bd9f-kube-api-access-sd28k\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:19.385315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.385154    5098 scope.go:122] "RemoveContainer" containerID="f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"
Oct 31 03:38:19.385866 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:19.385722    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624\": container with ID starting with f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624 not found: ID does not exist" containerID="f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"
Oct 31 03:38:19.385866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.385759    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624"} err="failed to get container status \"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624\": rpc error: code = NotFound desc = could not find container \"f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624\": container with ID starting with f9c2894a79f88c91cf937c2c5acd7df38dee490441b0ba225f446faecd9a2624 not found: ID does not exist"
Oct 31 03:38:19.385866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.385780    5098 scope.go:122] "RemoveContainer" containerID="0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"
Oct 31 03:38:19.386224 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:19.386192    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d\": container with ID starting with 0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d not found: ID does not exist" containerID="0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"
Oct 31 03:38:19.386322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.386227    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d"} err="failed to get container status \"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d\": rpc error: code = NotFound desc = could not find container \"0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d\": container with ID starting with 0ddee995fba27f5450b26e830fa2780e327364941159aa50069c01e0d43e294d not found: ID does not exist"
Oct 31 03:38:19.387569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.387546    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:38:19.394801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.394775    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-fc226c-openshift-default-5696b5c75c-qtrdd"]
Oct 31 03:38:19.627908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.627817    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" path="/var/lib/kubelet/pods/74f897d9-1668-4075-b31d-2129a857bd9f/volumes"
Oct 31 03:38:19.982984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:19.982909    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:38:21.016630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.016515    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.017585    5098 cpu_manager.go:414] "containerMap: removing container" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.017660    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.017754    5098 cpu_manager.go:414] "containerMap: removing container" podUID="71aa485d-c994-4af0-935b-ca22e1616723" containerName="revoke-keys"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.017790    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71aa485d-c994-4af0-935b-ca22e1616723" containerName="revoke-keys"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.018510    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.018538    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.018550    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="71aa485d-c994-4af0-935b-ca22e1616723" containerName="revoke-keys"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.027214    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.033947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.033729    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-619d46-openshift-default-dockercfg-n52s8\""
Oct 31 03:38:21.037353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.037326    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:38:21.087325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087274    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087343    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087411    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgxqb\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087620    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087663    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087773    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.087997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.087973    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.088930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.088099    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.088930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.088164    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.189918    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190030    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190094    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190178    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190211    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190277    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgxqb\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190455    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190502    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.191660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.190560    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.210640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.201637    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.240785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.214982    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.241046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.224558    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.241193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.232037    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.266109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.240642    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.316633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.312340    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgxqb\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.342649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.335000    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.342649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.339033    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.376631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.376145    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\") pod \"e2e-api-b-619d46-openshift-default-65646c4d47-f8x54\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") " pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:21.662085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:21.661681    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:22.263342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.263305    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:38:22.368355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.368316    5098 cpu_manager.go:414] "containerMap: removing container" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:22.368355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.368353    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f897d9-1668-4075-b31d-2129a857bd9f" containerName="istio-proxy"
Oct 31 03:38:22.412526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.412495    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:38:22.412858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.412550    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:38:22.412858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.412677    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.425744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.425304    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-8361df-openshift-default-dockercfg-n4cvz\""
Oct 31 03:38:22.428126 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:22.427667    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1fa7fd6_c485_42ff_9680_8d76dcad89bb.slice/crio-8828b052bf81e28a89e8b432aef599e1905e73a99a63a6ed43136b6da1f03096 WatchSource:0}: Error finding container 8828b052bf81e28a89e8b432aef599e1905e73a99a63a6ed43136b6da1f03096: Status 404 returned error can't find the container with id 8828b052bf81e28a89e8b432aef599e1905e73a99a63a6ed43136b6da1f03096
Oct 31 03:38:22.431095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.431065    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:22.431216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.431151    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:22.431216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.431199    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.570840    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.570905    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49xf\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.570949    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571006    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571055    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571260    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571300    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571408    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.586767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.571440    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.601726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.601683    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerStarted","Data":"8828b052bf81e28a89e8b432aef599e1905e73a99a63a6ed43136b6da1f03096"}
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687218    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687277    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687416    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687598    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687650    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49xf\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687687    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687751    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.687810    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.696626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.688335    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.721629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.713141    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.721629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.714338    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.721629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.720590    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.731861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.728644    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.746642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.744142    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.746642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.744918    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.746642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.745017    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.749712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.749219    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49xf\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\") pod \"e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") " pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:22.892393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:22.892094    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:23.228557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.228519    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:38:23.228738 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:23.228623    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e31255_64f6_419b_83a4_1a78222b96c8.slice/crio-45700e678ad5b2eb13ecb3925ee0c09f957ee9beeec4eeec6c06a7bf1778b09b WatchSource:0}: Error finding container 45700e678ad5b2eb13ecb3925ee0c09f957ee9beeec4eeec6c06a7bf1778b09b: Status 404 returned error can't find the container with id 45700e678ad5b2eb13ecb3925ee0c09f957ee9beeec4eeec6c06a7bf1778b09b
Oct 31 03:38:23.232487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.232447    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:23.232595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.232523    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:23.232595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.232558    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:23.661979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.661907    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerStarted","Data":"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"}
Oct 31 03:38:23.661979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.661973    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerStarted","Data":"45700e678ad5b2eb13ecb3925ee0c09f957ee9beeec4eeec6c06a7bf1778b09b"}
Oct 31 03:38:23.662461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.661998    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerStarted","Data":"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"}
Oct 31 03:38:23.666430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.666397    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:23.679986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.679950    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:23.688806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.688747    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podStartSLOduration=2.68872788 podStartE2EDuration="2.68872788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:23.68584086 +0000 UTC m=+1378.211060450" watchObservedRunningTime="2026-10-31 03:38:23.68872788 +0000 UTC m=+1378.213947469"
Oct 31 03:38:23.725826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.725770    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podStartSLOduration=3.725750571 podStartE2EDuration="3.725750571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:23.720837998 +0000 UTC m=+1378.246057590" watchObservedRunningTime="2026-10-31 03:38:23.725750571 +0000 UTC m=+1378.250970242"
Oct 31 03:38:23.907632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:23.906740    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:24.679686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.679581    5098 generic.go:356] "Generic (PLEG): container finished" podID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerID="1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e" exitCode=0
Oct 31 03:38:24.679686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.679655    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerDied","Data":"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"}
Oct 31 03:38:24.680644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.680580    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:24.680867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.680806    5098 scope.go:122] "RemoveContainer" containerID="1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"
Oct 31 03:38:24.684345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.684299    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:24.684475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.684399    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:24.684475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.684448    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:24.685918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:24.685440    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:38:25.199649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.192993    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:25.692215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.692179    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerStarted","Data":"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"}
Oct 31 03:38:25.692916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.692894    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:25.693721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.693700    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:38:25.719080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.719049    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:38:25.720805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.720770    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:25.720961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:25.720837    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:26.719886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:26.719847    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:26.720346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:26.719901    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:27.719971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:27.719934    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:27.720433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:27.720003    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:28.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:28.808092    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:28.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:28.808185    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:29.742634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:29.731751    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:29.742634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:29.731834    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:30.034237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:30.034199    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:38:30.735721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:30.727052    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:30.735721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:30.727131    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:31.090865 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:31.090811    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:38:31.719367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:31.719328    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:31.719553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:31.719405    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:32.106628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:32.097403    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"]
Oct 31 03:38:32.110753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:32.110689    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a50137d-skh2t"]
Oct 31 03:38:32.719359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:32.719308    5098 patch_prober.go:29] interesting pod/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body=
Oct 31 03:38:32.719557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:32.719388    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused"
Oct 31 03:38:33.615049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:33.615013    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="612204b8-10ee-4033-80ea-e1759e7e13d8" path="/var/lib/kubelet/pods/612204b8-10ee-4033-80ea-e1759e7e13d8/volumes"
Oct 31 03:38:34.225627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.191694    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:38:34.250658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.250580    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:38:34.305268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.305231    5098 scope.go:122] "RemoveContainer" containerID="d81cbcb2b335ce249017a1735310eac00e81d70a668ad222966c752bbcfd2503"
Oct 31 03:38:34.319101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.319072    5098 scope.go:122] "RemoveContainer" containerID="504d067eb4f327920d62a8500f8a1e6e7b644aeb4e61af39b64098d9afb7b0b9"
Oct 31 03:38:34.331058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.331034    5098 scope.go:122] "RemoveContainer" containerID="9946afa99e45658ea3b8934e09a4ef50f8422df48ba0d3e96dae8a8d646df51c"
Oct 31 03:38:34.345450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.345422    5098 scope.go:122] "RemoveContainer" containerID="23baa43d2c27c548e44ff2ea24207387fafcb7acaf29d785a548d7fa3f9d282d"
Oct 31 03:38:34.356888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.356859    5098 scope.go:122] "RemoveContainer" containerID="3597897894a8c837a72a1078ce3ec53941e8ac8bfc03ca86115753b207721840"
Oct 31 03:38:34.374391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.374368    5098 scope.go:122] "RemoveContainer" containerID="bcd6b02424c6d95503b9484cc15e94363f48955f52d31384a4bb6630858d482e"
Oct 31 03:38:34.388957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.388931    5098 scope.go:122] "RemoveContainer" containerID="6a5d27bedae344ea238c23290a5dad87caa2ed23d413eb49199f274d4a6de31a"
Oct 31 03:38:34.747144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.747044    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:38:34.747546 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.747308    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy" containerID="cri-o://3eb887d06510b6f74cea2b1d495f56e79c30d00766946cb511ec66a8bf21e4d9" gracePeriod=30
Oct 31 03:38:34.758696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.758575    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:38:34.758888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:34.758810    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy" containerID="cri-o://ea3173a898ad6b11e09f287d54d47c5f92f58ed8bf9b042bbf7a8b8897dbbcb7" gracePeriod=30
Oct 31 03:38:36.133647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:36.132691    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:38:36.133647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:36.132985    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy" containerID="cri-o://dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b" gracePeriod=30
Oct 31 03:38:36.140496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:36.139428    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:38:36.341552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:36.341496    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy" containerID="cri-o://4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9" gracePeriod=30
Oct 31 03:38:40.713686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:40.695100    5098 generic.go:356] "Generic (PLEG): container finished" podID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerID="3eb887d06510b6f74cea2b1d495f56e79c30d00766946cb511ec66a8bf21e4d9" exitCode=0
Oct 31 03:38:40.750414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:40.701715    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" event={"ID":"df91633e-99b9-43ce-a114-fecd7f1c3bb0","Type":"ContainerDied","Data":"3eb887d06510b6f74cea2b1d495f56e79c30d00766946cb511ec66a8bf21e4d9"}
Oct 31 03:38:40.943644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:40.932752    5098 generic.go:356] "Generic (PLEG): container finished" podID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerID="ea3173a898ad6b11e09f287d54d47c5f92f58ed8bf9b042bbf7a8b8897dbbcb7" exitCode=0
Oct 31 03:38:40.943644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:40.932883    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" event={"ID":"96cc0fd8-4dbc-414f-927c-ce58c7239566","Type":"ContainerDied","Data":"ea3173a898ad6b11e09f287d54d47c5f92f58ed8bf9b042bbf7a8b8897dbbcb7"}
Oct 31 03:38:40.958372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:40.957147    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:38:41.033055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.033017    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.046481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.040880    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.063811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.063774    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.064119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.064101    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.064503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.064484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.064686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.064669    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.064910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.064895    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.065050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.065035    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.065185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.065172    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\") pod \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\" (UID: \"df91633e-99b9-43ce-a114-fecd7f1c3bb0\") "
Oct 31 03:38:41.068044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.067994    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.083983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.068451    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:41.084177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.068789    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.084300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.068863    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.084404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.070226    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.091723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.091674    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "kube-api-access-cb4j5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.092054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.092030    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:41.096842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.095779    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:38:41.097564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.097519    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.097804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.097785    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy" pod "df91633e-99b9-43ce-a114-fecd7f1c3bb0" (UID: "df91633e-99b9-43ce-a114-fecd7f1c3bb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.167069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167041    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167157    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167180    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167303    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.167356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167320    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167414    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167459    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167576    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167624    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.167728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167693    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167771    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\") pod \"96cc0fd8-4dbc-414f-927c-ce58c7239566\" (UID: \"96cc0fd8-4dbc-414f-927c-ce58c7239566\") "
Oct 31 03:38:41.167918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167689    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:41.167918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167810    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.167918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.167858    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.168577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168555    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168579    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/96cc0fd8-4dbc-414f-927c-ce58c7239566-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168591    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168600    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168624    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168632    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168642    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168650    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168661    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168671    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168680    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168692    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb4j5\" (UniqueName: \"kubernetes.io/projected/df91633e-99b9-43ce-a114-fecd7f1c3bb0-kube-api-access-cb4j5\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168704    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.168711    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df91633e-99b9-43ce-a114-fecd7f1c3bb0-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.170598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.170567    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.170598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.170578    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "kube-api-access-tpsnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.170838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.170652    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.170838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.170732    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo" pod "96cc0fd8-4dbc-414f-927c-ce58c7239566" (UID: "96cc0fd8-4dbc-414f-927c-ce58c7239566"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:41.253246 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:41.253190    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1d2074_b249_4424_8c9e_0f4ca5551a68.slice/crio-dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e1d2074_b249_4424_8c9e_0f4ca5551a68.slice/crio-conmon-dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:38:41.270402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.270293    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpsnj\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-kube-api-access-tpsnj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.270402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.270323    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.270402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.270334    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.270402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.270343    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/96cc0fd8-4dbc-414f-927c-ce58c7239566-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.477754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.477717    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:38:41.575343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575307    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575382    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575421    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575550    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575639    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575681    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575679    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.575801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575792    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575824    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.575968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575869    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.575968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.575961    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\") pod \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\" (UID: \"6e1d2074-b249-4424-8c9e-0f4ca5551a68\") "
Oct 31 03:38:41.576131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.576066    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.576187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.576170    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:41.576319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.576280    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.577465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.577432    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.577587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.577471    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.577587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.577487    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.577587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.577503    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.577587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.577518    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.578990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.578943    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:41.578990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.578952    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.579163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.579059    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.579361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.579339    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr" pod "6e1d2074-b249-4424-8c9e-0f4ca5551a68" (UID: "6e1d2074-b249-4424-8c9e-0f4ca5551a68"). InnerVolumeSpecName "kube-api-access-cr9tr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.679197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.679166    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cr9tr\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-kube-api-access-cr9tr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.679197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.679195    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.679197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.679205    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.679430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.679215    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e1d2074-b249-4424-8c9e-0f4ca5551a68-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.680672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.680584    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:38:41.780023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.779988    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780092    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780118    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780205    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780234    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780277    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780325    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780500    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.780879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780530    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780567    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\") pod \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\" (UID: \"c8cf84fc-dded-4821-82d5-7ed29859c0bc\") "
Oct 31 03:38:41.780879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780565    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.780879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780738    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.781150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.780899    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.781150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781102    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:41.781865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781788    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.781865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781827    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.781865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781842    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.781865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781858    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.782178 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.781873    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.782723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.782692    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:38:41.782829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.782793    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:38:41.782915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.782900    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "kube-api-access-sbn6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.783131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.783114    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token" pod "c8cf84fc-dded-4821-82d5-7ed29859c0bc" (UID: "c8cf84fc-dded-4821-82d5-7ed29859c0bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:41.883548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.883513    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.883548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.883548    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.883793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.883567    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbn6p\" (UniqueName: \"kubernetes.io/projected/c8cf84fc-dded-4821-82d5-7ed29859c0bc-kube-api-access-sbn6p\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.883793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.883577    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c8cf84fc-dded-4821-82d5-7ed29859c0bc-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:41.944084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.943992    5098 generic.go:356] "Generic (PLEG): container finished" podID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerID="4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9" exitCode=0
Oct 31 03:38:41.944084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.944046    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"
Oct 31 03:38:41.944363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.944038    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" event={"ID":"c8cf84fc-dded-4821-82d5-7ed29859c0bc","Type":"ContainerDied","Data":"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"}
Oct 31 03:38:41.944363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.944186    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv" event={"ID":"c8cf84fc-dded-4821-82d5-7ed29859c0bc","Type":"ContainerDied","Data":"5c0913cc878ac0a666c7266845672064daefaf0c8e0435dd87dbb025cc1eaca5"}
Oct 31 03:38:41.944363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.944216    5098 scope.go:122] "RemoveContainer" containerID="4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"
Oct 31 03:38:41.946162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.946140    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"
Oct 31 03:38:41.946283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.946209    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf" event={"ID":"96cc0fd8-4dbc-414f-927c-ce58c7239566","Type":"ContainerDied","Data":"cc7b60608d4b6da96cf01472f4861fc401629a55ccc7e7d986b8a9f126602ec2"}
Oct 31 03:38:41.948265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.948233    5098 generic.go:356] "Generic (PLEG): container finished" podID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerID="dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b" exitCode=0
Oct 31 03:38:41.948384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.948285    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"
Oct 31 03:38:41.948384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.948309    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" event={"ID":"6e1d2074-b249-4424-8c9e-0f4ca5551a68","Type":"ContainerDied","Data":"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"}
Oct 31 03:38:41.948384 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.948361    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp" event={"ID":"6e1d2074-b249-4424-8c9e-0f4ca5551a68","Type":"ContainerDied","Data":"f109a854514b592bd2a61a8a7c85143a7da8c0ee71422bac789af330497e253e"}
Oct 31 03:38:41.950473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.950451    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"
Oct 31 03:38:41.950569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.950469    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr" event={"ID":"df91633e-99b9-43ce-a114-fecd7f1c3bb0","Type":"ContainerDied","Data":"0aa95c9ff90b0f2b690e57923f158034d18499ebcbc9a4f16e041b67c40a00c2"}
Oct 31 03:38:41.959872 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.959852    5098 scope.go:122] "RemoveContainer" containerID="4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"
Oct 31 03:38:41.960219 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:41.960193    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9\": container with ID starting with 4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9 not found: ID does not exist" containerID="4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"
Oct 31 03:38:41.960324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.960226    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9"} err="failed to get container status \"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9\": rpc error: code = NotFound desc = could not find container \"4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9\": container with ID starting with 4405715ac8e5fc3cb41064f4deb80a4442f43a95cca740fd3016b64224b36ab9 not found: ID does not exist"
Oct 31 03:38:41.960324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.960244    5098 scope.go:122] "RemoveContainer" containerID="ea3173a898ad6b11e09f287d54d47c5f92f58ed8bf9b042bbf7a8b8897dbbcb7"
Oct 31 03:38:41.973567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.973545    5098 scope.go:122] "RemoveContainer" containerID="dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"
Oct 31 03:38:41.981935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.981908    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:38:41.985947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.985925    5098 scope.go:122] "RemoveContainer" containerID="dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"
Oct 31 03:38:41.986275 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:38:41.986253    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b\": container with ID starting with dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b not found: ID does not exist" containerID="dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"
Oct 31 03:38:41.986324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.986283    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b"} err="failed to get container status \"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b\": rpc error: code = NotFound desc = could not find container \"dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b\": container with ID starting with dfa806497956f20d3a576c161359bcd24cb18af25408db4c99384acb34e77c0b not found: ID does not exist"
Oct 31 03:38:41.986324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.986301    5098 scope.go:122] "RemoveContainer" containerID="3eb887d06510b6f74cea2b1d495f56e79c30d00766946cb511ec66a8bf21e4d9"
Oct 31 03:38:41.996338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:41.996308    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-7dnkp"]
Oct 31 03:38:42.008946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.008908    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:38:42.016416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.016383    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-b6dkr"]
Oct 31 03:38:42.039838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.039790    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:38:42.048159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.048118    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-n4txv"]
Oct 31 03:38:42.060171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.060136    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:38:42.068085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:42.068053    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-whsjf"]
Oct 31 03:38:43.291675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.291633    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:38:43.292811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292787    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy"
Oct 31 03:38:43.292917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292831    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy"
Oct 31 03:38:43.292917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292867    5098 cpu_manager.go:414] "containerMap: removing container" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy"
Oct 31 03:38:43.292917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292876    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy"
Oct 31 03:38:43.292917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292911    5098 cpu_manager.go:414] "containerMap: removing container" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy"
Oct 31 03:38:43.293068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292919    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy"
Oct 31 03:38:43.293068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292967    5098 cpu_manager.go:414] "containerMap: removing container" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy"
Oct 31 03:38:43.293068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.292974    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy"
Oct 31 03:38:43.293154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.293127    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" containerName="istio-proxy"
Oct 31 03:38:43.293154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.293146    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" containerName="istio-proxy"
Oct 31 03:38:43.293217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.293158    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" containerName="istio-proxy"
Oct 31 03:38:43.293217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.293169    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" containerName="istio-proxy"
Oct 31 03:38:43.297756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.297733    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.311508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.311478    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-007d82-openshift-default-dockercfg-s9xtf\""
Oct 31 03:38:43.312859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.312834    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.420804    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.420974    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421047    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tn8j\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421079    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421160    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421412    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421505    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421558    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.421630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.421587    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551204    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551296    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551349    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551375    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551455    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551591    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.551681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551670    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tn8j\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.552183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551696    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.552183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.551780    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.572579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.554072    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.572579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.554958    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.572579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.555547    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.572579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.556233    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.572579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.556503    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.681628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.681520    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.729716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.717401    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.729716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.723038    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.729716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.723929    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tn8j\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\") pod \"e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") " pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:43.765678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.749697    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e1d2074-b249-4424-8c9e-0f4ca5551a68" path="/var/lib/kubelet/pods/6e1d2074-b249-4424-8c9e-0f4ca5551a68/volumes"
Oct 31 03:38:43.769633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.768005    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96cc0fd8-4dbc-414f-927c-ce58c7239566" path="/var/lib/kubelet/pods/96cc0fd8-4dbc-414f-927c-ce58c7239566/volumes"
Oct 31 03:38:43.769633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.768988    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8cf84fc-dded-4821-82d5-7ed29859c0bc" path="/var/lib/kubelet/pods/c8cf84fc-dded-4821-82d5-7ed29859c0bc/volumes"
Oct 31 03:38:43.769908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.769809    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df91633e-99b9-43ce-a114-fecd7f1c3bb0" path="/var/lib/kubelet/pods/df91633e-99b9-43ce-a114-fecd7f1c3bb0/volumes"
Oct 31 03:38:43.918089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:43.918009    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:44.300516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:44.300481    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:38:44.301857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:44.301818    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:44.301971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:44.301904    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:44.301971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:44.301950    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:45.090668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:45.090583    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerStarted","Data":"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"}
Oct 31 03:38:45.090895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:45.090675    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerStarted","Data":"292a5bbab6ec62d445ac5830f3bb3bdc7232ba664c3d2e1ef36972943b730675"}
Oct 31 03:38:45.127395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:45.127336    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podStartSLOduration=2.127319708 podStartE2EDuration="2.127319708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:45.120152794 +0000 UTC m=+1399.645372384" watchObservedRunningTime="2026-10-31 03:38:45.127319708 +0000 UTC m=+1399.652539297"
Oct 31 03:38:45.918509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:45.918476    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:45.925038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:45.925010    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:46.209544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:46.209451    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:46.210702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:46.210589    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:38:46.954627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:46.954437    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:38:46.961636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:46.961171    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:46.981566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:46.971330    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a3b986-openshift-default-dockercfg-xjs8b\""
Oct 31 03:38:47.000749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.000716    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:38:47.101302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101257    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101326    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101496    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101583    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101742    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101820    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.101919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.101871    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.102070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.102051    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dqpj\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.102184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.102164    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.203701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203662    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.203929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203777    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.203929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203809    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.203929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203839    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.203929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203885    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.203972    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204025    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204088    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204252    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dqpj\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204408    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204448    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204575    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.204798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.204739    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.205064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.205046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.206824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.206803    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.207171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.207149    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.214621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.214577    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.215368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.215343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dqpj\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\") pod \"e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") " pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:47.306631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:47.304946    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:48.434351 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:48.434311    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae WatchSource:0}: Error finding container 6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae: Status 404 returned error can't find the container with id 6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae
Oct 31 03:38:48.449265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:48.449215    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:48.449420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:48.449320    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:48.449420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:48.449364    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:38:48.457431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:48.449861    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:38:49.904225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:49.904171    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerStarted","Data":"1a849f443ed87d1e4dca9fba4ab6af5f2df1e1f5cd016c041c2016b9a5b8109b"}
Oct 31 03:38:49.941101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:49.941040    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerStarted","Data":"6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae"}
Oct 31 03:38:49.960424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:49.960389    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:38:50.218689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.207565    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:38:50.218689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.207906    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.366676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.311729    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-007d82\""
Oct 31 03:38:50.366676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.312973    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-007d82-dockercfg-66p8j\""
Oct 31 03:38:50.439945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.416384    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:38:50.568564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.544150    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtkt\" (UniqueName: \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.568564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.544220    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.568564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.547737    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podStartSLOduration=4.547705151 podStartE2EDuration="4.547705151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:49.958776873 +0000 UTC m=+1404.483996467" watchObservedRunningTime="2026-10-31 03:38:50.547705151 +0000 UTC m=+1405.072924742"
Oct 31 03:38:50.568564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.562143    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:38:50.568564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.562214    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:38:50.625881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.625843    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:38:50.626082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.625940    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:50.626082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.625971    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.673636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.669886    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.673849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.673804    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzlj\" (UniqueName: \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.674494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.674466    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtkt\" (UniqueName: \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.674728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.674703    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.675483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.675458    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.684483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.684346    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtkt\" (UniqueName: \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\") pod \"payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.787631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.787422    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzlj\" (UniqueName: \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.801818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.787881    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.801818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.789909    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.808452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.808420    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzlj\" (UniqueName: \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\") pod \"payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") " pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:50.924034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.923946    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:50.944228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:50.944197    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:51.287203 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:51.287172    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16624416_1283_4b78_a5a7_94e585500f52.slice/crio-15dea2d9c4b865b4df26f518650d87690e6d6d4aa75383979f7b666176b6a929 WatchSource:0}: Error finding container 15dea2d9c4b865b4df26f518650d87690e6d6d4aa75383979f7b666176b6a929: Status 404 returned error can't find the container with id 15dea2d9c4b865b4df26f518650d87690e6d6d4aa75383979f7b666176b6a929
Oct 31 03:38:51.288776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:51.288734    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:38:51.312539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:51.312508    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:51.316959 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:38:51.316929    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod749df0ce_821d_48d3_bffa_f520ccb899c6.slice/crio-f16bda7d64e1e9efb5981b81a824d7520333223251cb2d9bd43b2ca3523f9149 WatchSource:0}: Error finding container f16bda7d64e1e9efb5981b81a824d7520333223251cb2d9bd43b2ca3523f9149: Status 404 returned error can't find the container with id f16bda7d64e1e9efb5981b81a824d7520333223251cb2d9bd43b2ca3523f9149
Oct 31 03:38:51.319016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:51.318917    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:38:52.051500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.051457    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" event={"ID":"749df0ce-821d-48d3-bffa-f520ccb899c6","Type":"ContainerStarted","Data":"ad532efd8d13c970da36b88eb28e8772850c45b40dd3a82e84c8520a2d155227"}
Oct 31 03:38:52.052091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.051527    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" event={"ID":"749df0ce-821d-48d3-bffa-f520ccb899c6","Type":"ContainerStarted","Data":"f16bda7d64e1e9efb5981b81a824d7520333223251cb2d9bd43b2ca3523f9149"}
Oct 31 03:38:52.052858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.052830    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:52.091634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.086831    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" event={"ID":"16624416-1283-4b78-a5a7-94e585500f52","Type":"ContainerStarted","Data":"61827a51631f1cb51734c6476854c7b3243d12d969481e63946f43be2c1831b9"}
Oct 31 03:38:52.091634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.086899    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:52.091634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.086927    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:52.091634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.086948    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" event={"ID":"16624416-1283-4b78-a5a7-94e585500f52","Type":"ContainerStarted","Data":"15dea2d9c4b865b4df26f518650d87690e6d6d4aa75383979f7b666176b6a929"}
Oct 31 03:38:52.091634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.088298    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:38:52.125958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.118040    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" podStartSLOduration=3.118017398 podStartE2EDuration="3.118017398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:52.075785643 +0000 UTC m=+1406.601005236" watchObservedRunningTime="2026-10-31 03:38:52.118017398 +0000 UTC m=+1406.643236987"
Oct 31 03:38:52.166752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.160910    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" podStartSLOduration=3.160885565 podStartE2EDuration="3.160885565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:38:52.14614963 +0000 UTC m=+1406.671369221" watchObservedRunningTime="2026-10-31 03:38:52.160885565 +0000 UTC m=+1406.686105155"
Oct 31 03:38:52.540065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.540028    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:38:52.595834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:52.595791    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:38:57.706429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:57.706355    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"]
Oct 31 03:38:57.711808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:57.711775    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b87769-sqq7r"]
Oct 31 03:38:58.775079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:58.774923    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:38:58.775079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:58.775012    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:38:59.195728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.195666    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" podUID="749df0ce-821d-48d3-bffa-f520ccb899c6" containerName="payload-processing" containerID="cri-o://ad532efd8d13c970da36b88eb28e8772850c45b40dd3a82e84c8520a2d155227" gracePeriod=25
Oct 31 03:38:59.195950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.195602    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" podUID="16624416-1283-4b78-a5a7-94e585500f52" containerName="payload-pre-processing" containerID="cri-o://61827a51631f1cb51734c6476854c7b3243d12d969481e63946f43be2c1831b9" gracePeriod=25
Oct 31 03:38:59.370731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.370695    5098 generic.go:356] "Generic (PLEG): container finished" podID="749df0ce-821d-48d3-bffa-f520ccb899c6" containerID="ad532efd8d13c970da36b88eb28e8772850c45b40dd3a82e84c8520a2d155227" exitCode=0
Oct 31 03:38:59.370929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.370760    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" event={"ID":"749df0ce-821d-48d3-bffa-f520ccb899c6","Type":"ContainerDied","Data":"ad532efd8d13c970da36b88eb28e8772850c45b40dd3a82e84c8520a2d155227"}
Oct 31 03:38:59.373195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.373146    5098 generic.go:356] "Generic (PLEG): container finished" podID="16624416-1283-4b78-a5a7-94e585500f52" containerID="61827a51631f1cb51734c6476854c7b3243d12d969481e63946f43be2c1831b9" exitCode=0
Oct 31 03:38:59.373365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.373239    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" event={"ID":"16624416-1283-4b78-a5a7-94e585500f52","Type":"ContainerDied","Data":"61827a51631f1cb51734c6476854c7b3243d12d969481e63946f43be2c1831b9"}
Oct 31 03:38:59.618469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.618431    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71aa485d-c994-4af0-935b-ca22e1616723" path="/var/lib/kubelet/pods/71aa485d-c994-4af0-935b-ca22e1616723/volumes"
Oct 31 03:38:59.621071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.621045    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:38:59.624044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.624016    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:38:59.631295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.631262    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\" (UniqueName: \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\") pod \"749df0ce-821d-48d3-bffa-f520ccb899c6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") "
Oct 31 03:38:59.631637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.631590    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\") pod \"749df0ce-821d-48d3-bffa-f520ccb899c6\" (UID: \"749df0ce-821d-48d3-bffa-f520ccb899c6\") "
Oct 31 03:38:59.633083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.633052    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume" pod "749df0ce-821d-48d3-bffa-f520ccb899c6" (UID: "749df0ce-821d-48d3-bffa-f520ccb899c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:59.634839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.634813    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj" pod "749df0ce-821d-48d3-bffa-f520ccb899c6" (UID: "749df0ce-821d-48d3-bffa-f520ccb899c6"). InnerVolumeSpecName "kube-api-access-qfzlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:59.707126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.707029    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:38:59.708365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708333    5098 cpu_manager.go:414] "containerMap: removing container" podUID="16624416-1283-4b78-a5a7-94e585500f52" containerName="payload-pre-processing"
Oct 31 03:38:59.708365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708361    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16624416-1283-4b78-a5a7-94e585500f52" containerName="payload-pre-processing"
Oct 31 03:38:59.708671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708411    5098 cpu_manager.go:414] "containerMap: removing container" podUID="749df0ce-821d-48d3-bffa-f520ccb899c6" containerName="payload-processing"
Oct 31 03:38:59.708671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708419    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="749df0ce-821d-48d3-bffa-f520ccb899c6" containerName="payload-processing"
Oct 31 03:38:59.708671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708643    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="16624416-1283-4b78-a5a7-94e585500f52" containerName="payload-pre-processing"
Oct 31 03:38:59.708671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.708667    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="749df0ce-821d-48d3-bffa-f520ccb899c6" containerName="payload-processing"
Oct 31 03:38:59.713257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.713228    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.713399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.713336    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:38:59.733488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.733448    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\") pod \"16624416-1283-4b78-a5a7-94e585500f52\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") "
Oct 31 03:38:59.733690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.733638    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\" (UniqueName: \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\") pod \"16624416-1283-4b78-a5a7-94e585500f52\" (UID: \"16624416-1283-4b78-a5a7-94e585500f52\") "
Oct 31 03:38:59.733973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.733925    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume" pod "16624416-1283-4b78-a5a7-94e585500f52" (UID: "16624416-1283-4b78-a5a7-94e585500f52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:38:59.734284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.734254    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5ght\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.734424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.734387    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.734622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.734579    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.735394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.735370    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.735803 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.735778    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.735956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.735940    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.736082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.736065    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.736511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.736492    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/749df0ce-821d-48d3-bffa-f520ccb899c6-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:59.736842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.736823    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzlj\" (UniqueName: \"kubernetes.io/projected/749df0ce-821d-48d3-bffa-f520ccb899c6-kube-api-access-qfzlj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:59.736970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.736956    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16624416-1283-4b78-a5a7-94e585500f52-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:59.737368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.737337    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt" pod "16624416-1283-4b78-a5a7-94e585500f52" (UID: "16624416-1283-4b78-a5a7-94e585500f52"). InnerVolumeSpecName "kube-api-access-jjtkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.846973    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5ght\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847103    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847193    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847227    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847413    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847453    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847486    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.849299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.847783    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtkt\" (UniqueName: \"kubernetes.io/projected/16624416-1283-4b78-a5a7-94e585500f52-kube-api-access-jjtkt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:38:59.857092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.852090    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.857594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.857564    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.865492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.862764    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.876012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.875138    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.876012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.875244    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.876300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.876127    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:38:59.878104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:38:59.878053    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5ght\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bczj4\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:39:00.033844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.033802    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:39:00.387176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.387139    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6" event={"ID":"749df0ce-821d-48d3-bffa-f520ccb899c6","Type":"ContainerDied","Data":"f16bda7d64e1e9efb5981b81a824d7520333223251cb2d9bd43b2ca3523f9149"}
Oct 31 03:39:00.387374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.387213    5098 scope.go:122] "RemoveContainer" containerID="ad532efd8d13c970da36b88eb28e8772850c45b40dd3a82e84c8520a2d155227"
Oct 31 03:39:00.387374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.387148    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"
Oct 31 03:39:00.389218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.389196    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"
Oct 31 03:39:00.389351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.389195    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv" event={"ID":"16624416-1283-4b78-a5a7-94e585500f52","Type":"ContainerDied","Data":"15dea2d9c4b865b4df26f518650d87690e6d6d4aa75383979f7b666176b6a929"}
Oct 31 03:39:00.403370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.403347    5098 scope.go:122] "RemoveContainer" containerID="61827a51631f1cb51734c6476854c7b3243d12d969481e63946f43be2c1831b9"
Oct 31 03:39:00.417791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.417758    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:39:00.424445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.424415    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-007d82-cb7f984f9-bfwt6"]
Oct 31 03:39:00.440967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.440921    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:39:00.449145 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:39:00.449103    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ef938d9_71a9_4248_83b3_f2e5a204c28e.slice/crio-80055eddbad8eac786d42df2264227c5fe3c028f6f709048d70623ab5afa44d6 WatchSource:0}: Error finding container 80055eddbad8eac786d42df2264227c5fe3c028f6f709048d70623ab5afa44d6: Status 404 returned error can't find the container with id 80055eddbad8eac786d42df2264227c5fe3c028f6f709048d70623ab5afa44d6
Oct 31 03:39:00.449672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.449599    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-007d82-59cb6f4c8c-6n5pv"]
Oct 31 03:39:00.452579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.452540    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:00.452716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.452637    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:00.458185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:00.458158    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:39:01.073135 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:39:01.073032    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:39:01.430431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.430312    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" event={"ID":"3ef938d9-71a9-4248-83b3-f2e5a204c28e","Type":"ContainerStarted","Data":"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"}
Oct 31 03:39:01.430735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.430707    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" event={"ID":"3ef938d9-71a9-4248-83b3-f2e5a204c28e","Type":"ContainerStarted","Data":"80055eddbad8eac786d42df2264227c5fe3c028f6f709048d70623ab5afa44d6"}
Oct 31 03:39:01.436758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.436714    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:39:01.437709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.437680    5098 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bczj4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:39:01.442502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.442470    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:39:01.462410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.462262    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" podStartSLOduration=9.462237326 podStartE2EDuration="9.462237326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:38:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:01.455501244 +0000 UTC m=+1415.980720835" watchObservedRunningTime="2026-10-31 03:39:01.462237326 +0000 UTC m=+1415.987456916"
Oct 31 03:39:01.622929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.620462    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16624416-1283-4b78-a5a7-94e585500f52" path="/var/lib/kubelet/pods/16624416-1283-4b78-a5a7-94e585500f52/volumes"
Oct 31 03:39:01.622929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:01.621120    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="749df0ce-821d-48d3-bffa-f520ccb899c6" path="/var/lib/kubelet/pods/749df0ce-821d-48d3-bffa-f520ccb899c6/volumes"
Oct 31 03:39:02.456054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.456015    5098 generic.go:356] "Generic (PLEG): container finished" podID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerID="99be3808da0c557e1c783aafdaa5207d0e046a76dd3d17c92c9e6374076fccbb" exitCode=0
Oct 31 03:39:02.456693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.456089    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerDied","Data":"99be3808da0c557e1c783aafdaa5207d0e046a76dd3d17c92c9e6374076fccbb"}
Oct 31 03:39:02.456937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.456917    5098 scope.go:122] "RemoveContainer" containerID="99be3808da0c557e1c783aafdaa5207d0e046a76dd3d17c92c9e6374076fccbb"
Oct 31 03:39:02.457004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.456944    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:39:02.459386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.459349    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:02.459492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.459444    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:02.459533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:02.459490    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:03.468743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:03.468704    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerStarted","Data":"cc0f21ae1fe38a4b0da0060b40ec0407726b112b844dd84aa61f921dc40d000e"}
Oct 31 03:39:04.237029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:04.236994    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:39:04.237246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:04.237222    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:04.237293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:04.237270    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:05.236865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:05.236830    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:05.237278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:05.236921    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:06.360643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:06.349787    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:06.360643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:06.349875    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:06.522742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:06.522702    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:39:06.523363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:06.523313    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy" containerID="cri-o://7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3" gracePeriod=30
Oct 31 03:39:07.241791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:07.241746    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:07.242083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:07.242052    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:08.238501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:08.238460    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:08.239144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:08.239108    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:09.236676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:09.236594    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:09.236884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:09.236702    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:10.236259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:10.236224    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused" start-of-body=
Oct 31 03:39:10.236688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:10.236281    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": dial tcp 10.132.0.180:15021: connect: connection refused"
Oct 31 03:39:11.241549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:11.241515    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:39:11.242083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:11.242005    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:39:11.242947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:11.242926    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:39:12.427039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.426072    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:39:12.469927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.469846    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:39:12.470696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470680    5098 cpu_manager.go:414] "containerMap: removing container" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.470745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470697    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.470745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470725    5098 cpu_manager.go:414] "containerMap: removing container" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.470745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470730    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.470865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470855    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.470902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.470872    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerName="istio-proxy"
Oct 31 03:39:12.476664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.475085    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.491557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.486282    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c5dcb7-openshift-default-dockercfg-qblrc\""
Oct 31 03:39:12.500626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.495897    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:39:12.536373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536338    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536425    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536489    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536515    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536581    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536666    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.536794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.536766    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.537045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537029    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.537105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537076    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\") pod \"22b1a180-c76b-4a9d-861f-243e8ed39e70\" (UID: \"22b1a180-c76b-4a9d-861f-243e8ed39e70\") "
Oct 31 03:39:12.537519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537495    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.537670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537561    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.537743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537679    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.537873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537857    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.537929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537895    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsh7\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.537980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537937    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.538039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.537979    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.538159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.538145    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.538261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.538196    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.538857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.538829    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:12.539068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.539046    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:12.539282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.539258    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:12.539596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.539554    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:12.539841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.539816    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:12.541635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.541594    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "kube-api-access-l2569". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:12.541827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.541723    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:12.542255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.542228    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:39:12.543946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.543918    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo" pod "22b1a180-c76b-4a9d-861f-243e8ed39e70" (UID: "22b1a180-c76b-4a9d-861f-243e8ed39e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:39:12.640374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640335    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640400    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640487    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640668    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640702    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsh7\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640736    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640769    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.640951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.640922    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641019    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641057    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641080    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641241    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641281    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641300    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1a180-c76b-4a9d-861f-243e8ed39e70-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641314    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641339    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641353    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641366    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2569\" (UniqueName: \"kubernetes.io/projected/22b1a180-c76b-4a9d-861f-243e8ed39e70-kube-api-access-l2569\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641380    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641395    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641313    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.641475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641408    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1a180-c76b-4a9d-861f-243e8ed39e70-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:12.641850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.641663    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.644110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.644082    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.644782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.644759    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.650484 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.650450    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.650678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.650651    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsh7\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\") pod \"e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") " pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.829572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.829530    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:12.966643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.965870    5098 generic.go:356] "Generic (PLEG): container finished" podID="22b1a180-c76b-4a9d-861f-243e8ed39e70" containerID="7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3" exitCode=0
Oct 31 03:39:12.976130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.967180    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"
Oct 31 03:39:12.989436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.989027    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerDied","Data":"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"}
Oct 31 03:39:12.989436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.989149    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc" event={"ID":"22b1a180-c76b-4a9d-861f-243e8ed39e70","Type":"ContainerDied","Data":"c51705f8bfbb2214982c2b63ef5723c295a9423662c034a7b59557e1dc6a9935"}
Oct 31 03:39:12.989436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:12.989180    5098 scope.go:122] "RemoveContainer" containerID="7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"
Oct 31 03:39:13.173654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.173617    5098 scope.go:122] "RemoveContainer" containerID="1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"
Oct 31 03:39:13.184439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.182267    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:39:13.190801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.189163    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-b87769-openshift-default-5978d6fb5-fctjc"]
Oct 31 03:39:13.226029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.225706    5098 scope.go:122] "RemoveContainer" containerID="7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"
Oct 31 03:39:13.228946 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:39:13.226349    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3\": container with ID starting with 7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3 not found: ID does not exist" containerID="7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"
Oct 31 03:39:13.228946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.226390    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3"} err="failed to get container status \"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3\": rpc error: code = NotFound desc = could not find container \"7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3\": container with ID starting with 7abdfcf04ad11e1c95f163d271e9c3048218b3b47bfe43b2900cceb00e7711c3 not found: ID does not exist"
Oct 31 03:39:13.228946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.226415    5098 scope.go:122] "RemoveContainer" containerID="1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"
Oct 31 03:39:13.239974 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:39:13.239871    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e\": container with ID starting with 1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e not found: ID does not exist" containerID="1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"
Oct 31 03:39:13.239974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.239926    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e"} err="failed to get container status \"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e\": rpc error: code = NotFound desc = could not find container \"1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e\": container with ID starting with 1a66b386575332085dc274cba8d7a8db3b774b1fa9d360133f8f4176d7a59c6e not found: ID does not exist"
Oct 31 03:39:13.403870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.403784    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:39:13.411816 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:39:13.411783    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138d3b3c_cada_4754_9f08_25a848c726a5.slice/crio-91c510a5182bb560c56daecfa08ad141f8083e3ccd32a2eb790731b8ec6cdead WatchSource:0}: Error finding container 91c510a5182bb560c56daecfa08ad141f8083e3ccd32a2eb790731b8ec6cdead: Status 404 returned error can't find the container with id 91c510a5182bb560c56daecfa08ad141f8083e3ccd32a2eb790731b8ec6cdead
Oct 31 03:39:13.417792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.417754    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:13.417908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.417833    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:13.417908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.417868    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:13.672420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.672326    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b1a180-c76b-4a9d-861f-243e8ed39e70" path="/var/lib/kubelet/pods/22b1a180-c76b-4a9d-861f-243e8ed39e70/volumes"
Oct 31 03:39:13.979766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.979655    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerStarted","Data":"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"}
Oct 31 03:39:13.979766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:13.979732    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerStarted","Data":"91c510a5182bb560c56daecfa08ad141f8083e3ccd32a2eb790731b8ec6cdead"}
Oct 31 03:39:14.005102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:14.005008    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podStartSLOduration=7.004986958 podStartE2EDuration="7.004986958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:14.001781546 +0000 UTC m=+1428.527001148" watchObservedRunningTime="2026-10-31 03:39:14.004986958 +0000 UTC m=+1428.530206547"
Oct 31 03:39:14.831138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:14.831099    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:14.838008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:14.837981    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:14.989220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:14.989186    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:14.990280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:14.990258    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:39:18.279889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.279852    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"]
Oct 31 03:39:18.288448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.288418    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.296657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.296377    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"]
Oct 31 03:39:18.297563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.297531    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:39:18.299342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.298264    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:39:18.299637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.299619    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:39:18.299910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.299894    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:39:18.372350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.372301    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.372602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.372387    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.372602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.372572    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.474113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.474077    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.474330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.474124    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.474330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.474226    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.474966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.474943    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.476676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.476636    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.483259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.483233    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\") pod \"maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:18.625189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:18.625090    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:19.226647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:19.222487    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"]
Oct 31 03:39:20.136600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:20.136556    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx" event={"ID":"23b5695e-ca07-4eee-91db-81ee47c9b502","Type":"ContainerStarted","Data":"9890bed15cece313089768d65d02c8a919b464db55baafbab63c1c6372e9cb8c"}
Oct 31 03:39:20.136600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:20.136641    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx" event={"ID":"23b5695e-ca07-4eee-91db-81ee47c9b502","Type":"ContainerStarted","Data":"db64aa57d72e0070949c98d98619463dade9f749be1b71d58ff016325c3142a7"}
Oct 31 03:39:20.159864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:20.159806    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx" podStartSLOduration=2.159789823 podStartE2EDuration="2.159789823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:39:20.155732012 +0000 UTC m=+1434.680951604" watchObservedRunningTime="2026-10-31 03:39:20.159789823 +0000 UTC m=+1434.685009411"
Oct 31 03:39:21.147330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:21.147285    5098 generic.go:356] "Generic (PLEG): container finished" podID="23b5695e-ca07-4eee-91db-81ee47c9b502" containerID="9890bed15cece313089768d65d02c8a919b464db55baafbab63c1c6372e9cb8c" exitCode=0
Oct 31 03:39:21.147764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:21.147374    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx" event={"ID":"23b5695e-ca07-4eee-91db-81ee47c9b502","Type":"ContainerDied","Data":"9890bed15cece313089768d65d02c8a919b464db55baafbab63c1c6372e9cb8c"}
Oct 31 03:39:22.340829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.340802    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:22.448811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.448732    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\") pod \"23b5695e-ca07-4eee-91db-81ee47c9b502\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") "
Oct 31 03:39:22.448946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.448835    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\") pod \"23b5695e-ca07-4eee-91db-81ee47c9b502\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") "
Oct 31 03:39:22.448946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.448861    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\") pod \"23b5695e-ca07-4eee-91db-81ee47c9b502\" (UID: \"23b5695e-ca07-4eee-91db-81ee47c9b502\") "
Oct 31 03:39:22.449291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.449263    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca" pod "23b5695e-ca07-4eee-91db-81ee47c9b502" (UID: "23b5695e-ca07-4eee-91db-81ee47c9b502"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:39:22.449524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.449506    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.451011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.450988    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca" pod "23b5695e-ca07-4eee-91db-81ee47c9b502" (UID: "23b5695e-ca07-4eee-91db-81ee47c9b502"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:39:22.451148 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.451127    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz" pod "23b5695e-ca07-4eee-91db-81ee47c9b502" (UID: "23b5695e-ca07-4eee-91db-81ee47c9b502"). InnerVolumeSpecName "kube-api-access-bqtvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:39:22.551237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.551195    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqtvz\" (UniqueName: \"kubernetes.io/projected/23b5695e-ca07-4eee-91db-81ee47c9b502-kube-api-access-bqtvz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.551237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.551235    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23b5695e-ca07-4eee-91db-81ee47c9b502-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:39:22.672134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:22.672097    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:39:23.181778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:23.181750    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"
Oct 31 03:39:23.182196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:23.181816    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx" event={"ID":"23b5695e-ca07-4eee-91db-81ee47c9b502","Type":"ContainerDied","Data":"db64aa57d72e0070949c98d98619463dade9f749be1b71d58ff016325c3142a7"}
Oct 31 03:39:23.182378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:23.182360    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db64aa57d72e0070949c98d98619463dade9f749be1b71d58ff016325c3142a7"
Oct 31 03:39:30.827376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:30.827321    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:39:30.849527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:30.849486    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:39:31.053969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:31.053920    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:39:31.068038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:31.067772    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded"
Oct 31 03:39:31.441371 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:39:31.441313    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:39:34.485445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:34.485410    5098 scope.go:122] "RemoveContainer" containerID="b7b9e92228bc934febceb25942e7076ab4eec0b9f992787b7dda2d0f213a581b"
Oct 31 03:39:34.499992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:34.499969    5098 scope.go:122] "RemoveContainer" containerID="edb7b874f974ee2946f2aa18e3f6c01f890bd0f57aefd6c6779b37d5ce29bf4b"
Oct 31 03:39:34.513942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:34.513913    5098 scope.go:122] "RemoveContainer" containerID="c29157efef871d43bc3bc72b99feb54e2ee9b1732621664ef92f44ce951814a9"
Oct 31 03:39:37.051152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:37.051077    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"]
Oct 31 03:39:37.057902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:37.057865    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1892ca4-jdgs4"]
Oct 31 03:39:37.631755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:37.631687    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06bff794-a1e0-403b-99cb-d4696b5051d2" path="/var/lib/kubelet/pods/06bff794-a1e0-403b-99cb-d4696b5051d2/volumes"
Oct 31 03:39:39.048704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:39.046842    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"]
Oct 31 03:39:39.048704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:39.048017    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57285316-pmzjr"]
Oct 31 03:39:39.617289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:39.617240    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3b6cedf-b986-4fdf-b87f-276472caa079" path="/var/lib/kubelet/pods/e3b6cedf-b986-4fdf-b87f-276472caa079/volumes"
Oct 31 03:39:40.415437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:40.415396    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:42810->10.132.0.188:15021: read: connection reset by peer" start-of-body=
Oct 31 03:39:40.781257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:40.415461    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": read tcp 10.132.0.2:42810->10.132.0.188:15021: read: connection reset by peer"
Oct 31 03:39:40.781257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:40.725706    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": read tcp 10.132.0.2:52568->10.132.0.189:15021: read: connection reset by peer" start-of-body=
Oct 31 03:39:40.781257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:40.725774    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": read tcp 10.132.0.2:52568->10.132.0.189:15021: read: connection reset by peer"
Oct 31 03:39:41.201986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.201900    5098 generic.go:356] "Generic (PLEG): container finished" podID="84e31255-64f6-419b-83a4-1a78222b96c8" containerID="db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0" exitCode=0
Oct 31 03:39:41.201986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.201970    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerDied","Data":"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"}
Oct 31 03:39:41.202587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.202559    5098 scope.go:122] "RemoveContainer" containerID="db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"
Oct 31 03:39:41.204350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.204326    5098 generic.go:356] "Generic (PLEG): container finished" podID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerID="82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3" exitCode=0
Oct 31 03:39:41.204486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.204376    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerDied","Data":"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"}
Oct 31 03:39:41.204919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.204894    5098 scope.go:122] "RemoveContainer" containerID="82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"
Oct 31 03:39:41.205363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.205265    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:41.205363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.205355    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:41.205510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.205398    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:41.206812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.206784    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:41.206926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.206865    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:41.206926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:41.206916    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:42.217770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.217734    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerStarted","Data":"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"}
Oct 31 03:39:42.220106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.220079    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerStarted","Data":"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"}
Oct 31 03:39:42.669097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.669058    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:39:42.669476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.669448    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:42.669667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.669641    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:42.899690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.893069    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:39:42.899690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.893202    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:42.899690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:42.893245    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:43.677545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:43.677066    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:43.677545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:43.677134    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:43.897928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:43.897427    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:43.897928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:43.897493    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:44.305305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.305270    5098 generic.go:356] "Generic (PLEG): container finished" podID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerID="f22a7bd9d840ba3334a8934b95c637a033d58ad67d90146517d1d488d97955e2" exitCode=0
Oct 31 03:39:44.305498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.305342    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerDied","Data":"f22a7bd9d840ba3334a8934b95c637a033d58ad67d90146517d1d488d97955e2"}
Oct 31 03:39:44.305934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.305919    5098 scope.go:122] "RemoveContainer" containerID="f22a7bd9d840ba3334a8934b95c637a033d58ad67d90146517d1d488d97955e2"
Oct 31 03:39:44.308225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.308191    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:44.309030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.308270    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:44.309030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.308301    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:39:44.681754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.669057    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:44.681754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.669120    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:44.801205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.801118    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:39:44.911634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.896102    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:44.911634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:44.896171    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:45.324720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:45.324683    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerStarted","Data":"1f0036969b3a8454ae2fe8df6baf9bc608eba8765aa46ee14bbd7d1bcf27c809"}
Oct 31 03:39:45.722247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:45.722144    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:45.722900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:45.722851    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:45.913633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:45.910659    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:45.913633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:45.910720    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:46.193633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.161152    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:39:46.193633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.161384    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:46.193633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.161447    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:46.679684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.671039    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:46.679684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.671106    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:46.978697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.978057    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:46.978697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:46.978127    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:47.240687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.236251    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:47.240687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.236311    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:47.672534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.672497    5098 patch_prober.go:29] interesting pod/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body=
Oct 31 03:39:47.672867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.672832    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused"
Oct 31 03:39:47.897066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.897023    5098 patch_prober.go:29] interesting pod/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Oct 31 03:39:47.897396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:47.897366    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Oct 31 03:39:48.073846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:48.070031    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:48.073846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:48.070096    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:48.854695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:48.854652    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:39:48.886501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:48.886471    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:39:48.942132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:48.941774    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:39:49.086480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:49.075015    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:49.086480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:49.075084    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:49.086480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:49.075714    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:39:49.086480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:49.077448    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:39:49.104520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:49.102500    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:39:50.069632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:50.068078    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:50.069632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:50.068147    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:51.074756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:51.074706    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:51.075315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:51.074778    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:52.113096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:52.113059    5098 patch_prober.go:29] interesting pod/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body=
Oct 31 03:39:52.113721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:52.113688    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused"
Oct 31 03:39:53.073264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:53.073231    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:39:53.073776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:53.073746    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:39:53.074493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:53.074471    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:39:59.305254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.305213    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:39:59.305812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.305472    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy" containerID="cri-o://1f0036969b3a8454ae2fe8df6baf9bc608eba8765aa46ee14bbd7d1bcf27c809" gracePeriod=30
Oct 31 03:39:59.918532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.918498    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"]
Oct 31 03:39:59.921378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.921347    5098 cpu_manager.go:414] "containerMap: removing container" podUID="23b5695e-ca07-4eee-91db-81ee47c9b502" containerName="revoke-keys"
Oct 31 03:39:59.921378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.921373    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23b5695e-ca07-4eee-91db-81ee47c9b502" containerName="revoke-keys"
Oct 31 03:39:59.921899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.921645    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="23b5695e-ca07-4eee-91db-81ee47c9b502" containerName="revoke-keys"
Oct 31 03:39:59.927552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.927525    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"]
Oct 31 03:39:59.927814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.927699    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:39:59.933343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.931918    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:39:59.933343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.933024    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:39:59.942119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.942086    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:39:59.942286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.942238    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:39:59.987672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.987593    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:39:59.987896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.987873    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hnb\" (UniqueName: \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:39:59.988313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:39:59.988201    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.090039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.089997    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.090253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.090127    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hnb\" (UniqueName: \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.090331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.090280    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.090744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.090721    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.092811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.092788    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.098581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.098552    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hnb\" (UniqueName: \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\") pod \"maas-api-revoke-keys-e2e-api-b-8361df-dblln\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.248945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.248854    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:00.549152 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:40:00.549120    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod560fe01d_e6c2_4262_9017_b182b4fea1b7.slice/crio-d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8 WatchSource:0}: Error finding container d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8: Status 404 returned error can't find the container with id d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8
Oct 31 03:40:00.550394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.550368    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"]
Oct 31 03:40:00.745577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:00.745538    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln" event={"ID":"560fe01d-e6c2-4262-9017-b182b4fea1b7","Type":"ContainerStarted","Data":"d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8"}
Oct 31 03:40:01.060245 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:40:01.060210    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:40:01.758408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:01.758370    5098 generic.go:356] "Generic (PLEG): container finished" podID="560fe01d-e6c2-4262-9017-b182b4fea1b7" containerID="3cd7ff93f3adefa844f5d04aa0e80200cebddd5847a971d59697a725912d9161" exitCode=0
Oct 31 03:40:01.758881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:01.758451    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln" event={"ID":"560fe01d-e6c2-4262-9017-b182b4fea1b7","Type":"ContainerDied","Data":"3cd7ff93f3adefa844f5d04aa0e80200cebddd5847a971d59697a725912d9161"}
Oct 31 03:40:02.978132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:02.978101    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:03.099872 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.099832    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\" (UniqueName: \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\") pod \"560fe01d-e6c2-4262-9017-b182b4fea1b7\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") "
Oct 31 03:40:03.100068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.100058    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\") pod \"560fe01d-e6c2-4262-9017-b182b4fea1b7\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") "
Oct 31 03:40:03.100163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.100150    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\") pod \"560fe01d-e6c2-4262-9017-b182b4fea1b7\" (UID: \"560fe01d-e6c2-4262-9017-b182b4fea1b7\") "
Oct 31 03:40:03.100452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.100420    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca" pod "560fe01d-e6c2-4262-9017-b182b4fea1b7" (UID: "560fe01d-e6c2-4262-9017-b182b4fea1b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:03.100752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.100733    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:03.102175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.102150    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb" pod "560fe01d-e6c2-4262-9017-b182b4fea1b7" (UID: "560fe01d-e6c2-4262-9017-b182b4fea1b7"). InnerVolumeSpecName "kube-api-access-f7hnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:03.102261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.102197    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca" pod "560fe01d-e6c2-4262-9017-b182b4fea1b7" (UID: "560fe01d-e6c2-4262-9017-b182b4fea1b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:03.203181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.203137    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/560fe01d-e6c2-4262-9017-b182b4fea1b7-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:03.203181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.203176    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7hnb\" (UniqueName: \"kubernetes.io/projected/560fe01d-e6c2-4262-9017-b182b4fea1b7-kube-api-access-f7hnb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:03.780071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.780033    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"
Oct 31 03:40:03.780071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.780053    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln" event={"ID":"560fe01d-e6c2-4262-9017-b182b4fea1b7","Type":"ContainerDied","Data":"d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8"}
Oct 31 03:40:03.780299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:03.780118    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d111f890b55ecdcc3263576509cc5267f1dffaaf81eac1b52419faec51111bf8"
Oct 31 03:40:04.790801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:04.790768    5098 generic.go:356] "Generic (PLEG): container finished" podID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerID="1f0036969b3a8454ae2fe8df6baf9bc608eba8765aa46ee14bbd7d1bcf27c809" exitCode=0
Oct 31 03:40:04.791243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:04.790838    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerDied","Data":"1f0036969b3a8454ae2fe8df6baf9bc608eba8765aa46ee14bbd7d1bcf27c809"}
Oct 31 03:40:04.791243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:04.790903    5098 scope.go:122] "RemoveContainer" containerID="f22a7bd9d840ba3334a8934b95c637a033d58ad67d90146517d1d488d97955e2"
Oct 31 03:40:05.107577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.107543    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:40:05.149893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.149855    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:40:05.150968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.150940    5098 cpu_manager.go:414] "containerMap: removing container" podUID="560fe01d-e6c2-4262-9017-b182b4fea1b7" containerName="revoke-keys"
Oct 31 03:40:05.150968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.150964    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="560fe01d-e6c2-4262-9017-b182b4fea1b7" containerName="revoke-keys"
Oct 31 03:40:05.151189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151010    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:05.151189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151018    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:05.151189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151037    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:05.151189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151045    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:05.151390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151307    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:05.151390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.151334    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="560fe01d-e6c2-4262-9017-b182b4fea1b7" containerName="revoke-keys"
Oct 31 03:40:05.155662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.155583    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.161782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.161742    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:40:05.162196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.161978    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 03:40:05.162196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.162161    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c5dcb7-serving-cert\""
Oct 31 03:40:05.162544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.162247    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:40:05.162544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.162304    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\""
Oct 31 03:40:05.173739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.173706    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:40:05.220797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.220756    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.220997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.220819    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.220997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.220860    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.220997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.220983    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.221172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221071    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.221172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221127    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.221261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221213    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.221327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221301    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:05.221377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221333    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.221377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221346    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:05.221472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221390    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:05.221472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221382    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:05.221472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221396    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:05.221472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.221410    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\") pod \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\" (UID: \"8fbdd1c8-3092-44b3-80be-7ff950f44a86\") "
Oct 31 03:40:05.222135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.222098    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.222267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.222171    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrbw\" (UniqueName: \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.222566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.222544    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.222809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.222778    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.223235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223130    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.223235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223159    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.223235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223174    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.223235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223188    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.223235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223204    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.223569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223473    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:40:05.223734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223700    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:05.223881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223864    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "kube-api-access-w7k2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:05.223983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.223963    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token" pod "8fbdd1c8-3092-44b3-80be-7ff950f44a86" (UID: "8fbdd1c8-3092-44b3-80be-7ff950f44a86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:05.325061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.324953    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.325243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325076    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.325243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325199    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.325373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325241    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrbw\" (UniqueName: \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.325479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325465    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.325552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325482    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7k2c\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-kube-api-access-w7k2c\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.325552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325493    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.325552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.325502    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fbdd1c8-3092-44b3-80be-7ff950f44a86-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:05.326349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.326315    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.328171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.328117    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.328959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.328935    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.335379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.335343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrbw\" (UniqueName: \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\") pod \"maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:05.483645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:05.481981    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:06.192115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.180482    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q" event={"ID":"8fbdd1c8-3092-44b3-80be-7ff950f44a86","Type":"ContainerDied","Data":"fd71b801854a6ac307812b061569e636d1b6297c64f85d22657a60431f63ebd8"}
Oct 31 03:40:06.192115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.180585    5098 scope.go:122] "RemoveContainer" containerID="1f0036969b3a8454ae2fe8df6baf9bc608eba8765aa46ee14bbd7d1bcf27c809"
Oct 31 03:40:06.192115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.180769    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"
Oct 31 03:40:06.427047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.426938    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:40:06.618626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.608103    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-555bd6-openshift-default-7f57586f99-f995q"]
Oct 31 03:40:06.884165 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:40:06.884121    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b0fb44_1dac_45dd_aae3_7919258ff8a6.slice/crio-2a332b97f24d12337e9b67b0f4567cd0978a8ab80875cf360a4bb024b885680a WatchSource:0}: Error finding container 2a332b97f24d12337e9b67b0f4567cd0978a8ab80875cf360a4bb024b885680a: Status 404 returned error can't find the container with id 2a332b97f24d12337e9b67b0f4567cd0978a8ab80875cf360a4bb024b885680a
Oct 31 03:40:06.886458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.886433    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:40:06.887450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:06.887421    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:40:07.222219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:07.222121    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" event={"ID":"55b0fb44-1dac-45dd-aae3-7919258ff8a6","Type":"ContainerStarted","Data":"2a332b97f24d12337e9b67b0f4567cd0978a8ab80875cf360a4bb024b885680a"}
Oct 31 03:40:07.640282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:07.640248    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" path="/var/lib/kubelet/pods/8fbdd1c8-3092-44b3-80be-7ff950f44a86/volumes"
Oct 31 03:40:12.645822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:12.645735    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"]
Oct 31 03:40:12.724662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:12.721882    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-adb7b7-rjqqx"]
Oct 31 03:40:13.085219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:13.085061    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:13.642542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:13.642498    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23b5695e-ca07-4eee-91db-81ee47c9b502" path="/var/lib/kubelet/pods/23b5695e-ca07-4eee-91db-81ee47c9b502/volumes"
Oct 31 03:40:16.031885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:16.031836    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:16.032374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:16.031914    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:16.524977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:16.524923    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" event={"ID":"55b0fb44-1dac-45dd-aae3-7919258ff8a6","Type":"ContainerStarted","Data":"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"}
Oct 31 03:40:16.525191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:16.525019    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:16.546906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:16.546834    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" podStartSLOduration=28.647097365 podStartE2EDuration="37.546812675s" totalImagesPullingTime="8.89971531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:39:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:16.544421007 +0000 UTC m=+1491.069640595" watchObservedRunningTime="2026-10-31 03:40:16.546812675 +0000 UTC m=+1491.072032265"
Oct 31 03:40:17.250877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:17.250836    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:17.251393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:17.250913    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:20.061066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.061023    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"]
Oct 31 03:40:20.089628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.072562    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acc26b59-swvjb"]
Oct 31 03:40:20.089628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.086353    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:40:20.601002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.600908    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" containerID="cri-o://cc0f21ae1fe38a4b0da0060b40ec0407726b112b844dd84aa61f921dc40d000e" gracePeriod=30
Oct 31 03:40:20.681068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.681034    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"]
Oct 31 03:40:20.682096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.682070    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fbdd1c8-3092-44b3-80be-7ff950f44a86" containerName="istio-proxy"
Oct 31 03:40:20.724077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.724032    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"]
Oct 31 03:40:20.724273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.724189    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:20.763772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.763733    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:40:20.763970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.763846    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:40:20.904457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.883263    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:20.904457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.883446    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mlsw\" (UniqueName: \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:20.904457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:20.883502    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.001549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.001497    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.002188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.002153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mlsw\" (UniqueName: \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.002395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.002374    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.003397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.003363    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.015734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.015589    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.044512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.044424    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mlsw\" (UniqueName: \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\") pod \"maas-api-revoke-keys-e2e-api-b-619d46-7fxd9\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.062828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.062125    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:21.393083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.393049    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"]
Oct 31 03:40:21.393332 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:40:21.393303    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df57478_c4d2_4c36_aad4_b3d43d3737a5.slice/crio-0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84 WatchSource:0}: Error finding container 0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84: Status 404 returned error can't find the container with id 0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84
Oct 31 03:40:21.624685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.624649    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fed25ec-c903-4c58-b647-43aedfb5109b" path="/var/lib/kubelet/pods/8fed25ec-c903-4c58-b647-43aedfb5109b/volumes"
Oct 31 03:40:21.625078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:21.625058    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9" event={"ID":"9df57478-c4d2-4c36-aad4-b3d43d3737a5","Type":"ContainerStarted","Data":"0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84"}
Oct 31 03:40:22.545540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:22.545510    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:40:22.630354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:22.630319    5098 generic.go:356] "Generic (PLEG): container finished" podID="9df57478-c4d2-4c36-aad4-b3d43d3737a5" containerID="6a56a0f220f3a9a2a3759a6432b0e76efeb3820aaafcb641c9bc77920f3f96d0" exitCode=0
Oct 31 03:40:22.630540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:22.630407    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9" event={"ID":"9df57478-c4d2-4c36-aad4-b3d43d3737a5","Type":"ContainerDied","Data":"6a56a0f220f3a9a2a3759a6432b0e76efeb3820aaafcb641c9bc77920f3f96d0"}
Oct 31 03:40:23.136253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.136209    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:23.136465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.136291    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:23.764634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.764345    5098 generic.go:356] "Generic (PLEG): container finished" podID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerID="93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2" exitCode=0
Oct 31 03:40:23.764634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.764524    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerDied","Data":"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"}
Oct 31 03:40:23.774630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.772526    5098 generic.go:356] "Generic (PLEG): container finished" podID="138d3b3c-cada-4754-9f08-25a848c726a5" containerID="65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485" exitCode=0
Oct 31 03:40:23.774630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.772812    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerDied","Data":"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"}
Oct 31 03:40:23.789631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.778056    5098 scope.go:122] "RemoveContainer" containerID="93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"
Oct 31 03:40:23.789631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.779763    5098 scope.go:122] "RemoveContainer" containerID="65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"
Oct 31 03:40:23.805250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.804934    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.805250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.805046    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.805250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.805093    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.822494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.820971    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.822494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.821094    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.822494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.821148    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.924412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.894647    5098 generic.go:356] "Generic (PLEG): container finished" podID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerID="1a849f443ed87d1e4dca9fba4ab6af5f2df1e1f5cd016c041c2016b9a5b8109b" exitCode=0
Oct 31 03:40:23.924412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.894971    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerDied","Data":"1a849f443ed87d1e4dca9fba4ab6af5f2df1e1f5cd016c041c2016b9a5b8109b"}
Oct 31 03:40:23.924412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.895545    5098 scope.go:122] "RemoveContainer" containerID="1a849f443ed87d1e4dca9fba4ab6af5f2df1e1f5cd016c041c2016b9a5b8109b"
Oct 31 03:40:23.952098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.952054    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.952464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.952433    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:23.952688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:23.952667    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:24.407637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.406438    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:24.526168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.526131    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\" (UniqueName: \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\") pod \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") "
Oct 31 03:40:24.526385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.526199    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\") pod \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") "
Oct 31 03:40:24.526448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.526403    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\") pod \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\" (UID: \"9df57478-c4d2-4c36-aad4-b3d43d3737a5\") "
Oct 31 03:40:24.527023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.526993    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca" pod "9df57478-c4d2-4c36-aad4-b3d43d3737a5" (UID: "9df57478-c4d2-4c36-aad4-b3d43d3737a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:24.527408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.527385    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:24.529736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.529531    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw" pod "9df57478-c4d2-4c36-aad4-b3d43d3737a5" (UID: "9df57478-c4d2-4c36-aad4-b3d43d3737a5"). InnerVolumeSpecName "kube-api-access-9mlsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:24.529860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.529774    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca" pod "9df57478-c4d2-4c36-aad4-b3d43d3737a5" (UID: "9df57478-c4d2-4c36-aad4-b3d43d3737a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:24.629168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.629063    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mlsw\" (UniqueName: \"kubernetes.io/projected/9df57478-c4d2-4c36-aad4-b3d43d3737a5-kube-api-access-9mlsw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:24.629168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:24.629096    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df57478-c4d2-4c36-aad4-b3d43d3737a5-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:25.027688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.024628    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9" event={"ID":"9df57478-c4d2-4c36-aad4-b3d43d3737a5","Type":"ContainerDied","Data":"0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84"}
Oct 31 03:40:25.027688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.024712    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a481182f62f072d72e5da2729662adcd719b59f487c73ae2a456cc160ebbf84"
Oct 31 03:40:25.027688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.024797    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"
Oct 31 03:40:25.043673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.041227    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerStarted","Data":"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"}
Oct 31 03:40:25.072224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.072182    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerStarted","Data":"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"}
Oct 31 03:40:25.089661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.089623    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerStarted","Data":"b237f2136ca1ba0fa537a49af6541d292fcf4e3da904134d00b6106d46301373"}
Oct 31 03:40:25.306830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.306725    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:40:25.306830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.306794    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:25.307071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.306866    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:25.830884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.830793    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:40:25.831441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.831406    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:25.831591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.831457    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:25.945274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.943808    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:40:25.946213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.946178    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:25.952682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:25.946416    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:26.769289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.769250    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:26.769977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.769923    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:26.873631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.873328    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:26.873631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.873389    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:26.937674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.935528    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:26.937674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.935657    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:26.957633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.946662    5098 generic.go:356] "Generic (PLEG): container finished" podID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerID="cc0f21ae1fe38a4b0da0060b40ec0407726b112b844dd84aa61f921dc40d000e" exitCode=0
Oct 31 03:40:26.957633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.947967    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerDied","Data":"cc0f21ae1fe38a4b0da0060b40ec0407726b112b844dd84aa61f921dc40d000e"}
Oct 31 03:40:26.957633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.948034    5098 scope.go:122] "RemoveContainer" containerID="99be3808da0c557e1c783aafdaa5207d0e046a76dd3d17c92c9e6374076fccbb"
Oct 31 03:40:26.995080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:26.994964    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:40:27.143824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.143777    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.146790    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.146867    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.146919    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.147156    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.147201    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.147378    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.147413    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.177994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.147472    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\") pod \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\" (UID: \"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f\") "
Oct 31 03:40:27.202644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.199598    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:27.202644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.202064    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:27.202644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.202340    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:27.202644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.202508    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.206755    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.255115    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.255150    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.255168    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.255183    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.265820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.255200    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.286719    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.286800    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.286841    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "kube-api-access-9qnmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.298325    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo" pod "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" (UID: "5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.306089    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.306163    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.325047    5098 patch_prober.go:29] interesting pod/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:27.354046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.325730    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.180:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:27.371636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.366320    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qnmb\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-kube-api-access-9qnmb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.371636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.366367    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.371636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.366381    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.371636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.366394    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.542631    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.543951    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9df57478-c4d2-4c36-aad4-b3d43d3737a5" containerName="revoke-keys"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.543976    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df57478-c4d2-4c36-aad4-b3d43d3737a5" containerName="revoke-keys"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544061    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544071    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544092    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544101    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544265    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544281    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" containerName="istio-proxy"
Oct 31 03:40:27.544899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.544299    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df57478-c4d2-4c36-aad4-b3d43d3737a5" containerName="revoke-keys"
Oct 31 03:40:27.599596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.598300    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.599596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.624980    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-880e04-serving-cert\""
Oct 31 03:40:27.723645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.723578    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:40:27.747102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.747064    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmbhg\" (UniqueName: \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.817051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.789527    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.841966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.790756    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.841966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.827065    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.841966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.832081    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:27.841966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.832146    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:27.938989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.938773    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:27.938989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.938837    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:27.939311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.939153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.939311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.939198    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.939461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.939380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmbhg\" (UniqueName: \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.939461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.939442    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.940471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.940343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.942623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.942586    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:27.945019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:27.944990    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:28.028187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.026500    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmbhg\" (UniqueName: \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\") pod \"maas-api-e2e-inf-b-880e04-67cf98484-ntxfz\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:28.047990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.047960    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"
Oct 31 03:40:28.048342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.047963    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2" event={"ID":"5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f","Type":"ContainerDied","Data":"575abf8b69de795b9305a313f67efa0149be8ce6d8e9f0f1fe74f41d43ddb8d2"}
Oct 31 03:40:28.048424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.048392    5098 scope.go:122] "RemoveContainer" containerID="cc0f21ae1fe38a4b0da0060b40ec0407726b112b844dd84aa61f921dc40d000e"
Oct 31 03:40:28.074425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.073429    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:28.344714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.343502    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:40:28.345457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.345030    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:28.345457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.345088    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:28.393217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.393180    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-adb7b7-openshift-default-754fbdc97f-txqv2"]
Oct 31 03:40:28.852661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.838117    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:28.852661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.838181    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:28.933449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.931255    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:28.933449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:28.931314    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:29.554205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.553714    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:29.554205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.553775    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:29.831577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.831464    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:29.897996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.889489    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:40:29.949476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.923302    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:40:29.949476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.923524    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:40:29.977651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.956940    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:29.977651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.957012    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:29.977651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.974157    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f" path="/var/lib/kubelet/pods/5aa540e6-e2f3-4e23-aa12-ae4b2d39b63f/volumes"
Oct 31 03:40:29.977979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:29.977719    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:40:30.012515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.003688    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:30.012515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.003776    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:30.012515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.005722    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:40:30.012515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.006874    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:30.093698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.080083    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:30.250637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.234185    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:40:30.317664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.314180    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:30.317664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.314244    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:30.804382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.804290    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:40:30.838866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.838832    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:30.840795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.840757    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:30.933225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.933186    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:40:30.949380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.949347    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:30.952815 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.952670    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:30.954474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.953969    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:40:30.956252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:30.956226    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:40:31.084842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.015921    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:31.084842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.016524    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:40:31.388693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.384379    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:31.388693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.384450    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:31.439035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.435541    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:40:31.443645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:31.443464    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:40:32.105631 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:40:32.097384    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:40:32.160030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.159869    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:32.160030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.159957    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:32.168823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.168774    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:32.169161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.169121    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:32.218218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.218091    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Oct 31 03:40:32.335964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.335913    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:32.337668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.337624    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:32.602634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.570810    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"]
Oct 31 03:40:32.694685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.694133    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"]
Oct 31 03:40:32.694685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.694303    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:40:32.694685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.694330    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" event={"ID":"4bff431f-28d7-43a6-9181-e7333aceb93e","Type":"ContainerStarted","Data":"d7432eb2651f931b987a3355a64cd64a582e476b5557f33d029e4b303dc56822"}
Oct 31 03:40:32.694685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.694431    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.714817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.714786    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:40:32.756632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.735908    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:40:32.820733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.814366    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.820733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.814457    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.820733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.814549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.836650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.832257    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body=
Oct 31 03:40:32.836650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.832378    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused"
Oct 31 03:40:32.893939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.879478    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:32.935637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.934577    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.935637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.934680    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.935637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.934771    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.935967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.935921    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:32.982689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:32.974705    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:40:33.083221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.073122    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:33.083221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.073540    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:33.083221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.073589    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:33.094998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.093428    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:33.144339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.143457    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:33.214635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.213734    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:33.275327 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.275236    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-74c760-xswdl\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:33.325837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.325801    5098 patch_prober.go:29] interesting pod/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused" start-of-body=
Oct 31 03:40:33.326138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.326103    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.191:15021/healthz/ready\": dial tcp 10.132.0.191:15021: connect: connection refused"
Oct 31 03:40:33.336972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.336712    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:33.445761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.445719    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" event={"ID":"4bff431f-28d7-43a6-9181-e7333aceb93e","Type":"ContainerStarted","Data":"7329dabf3a0035affe5b28fa28cbffb68cb0a1ee3836906970c3a7e1ba3115a3"}
Oct 31 03:40:33.446351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.446323    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:33.528429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.528375    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" podStartSLOduration=14.545242899 podStartE2EDuration="16.528356463s" totalImagesPullingTime="1.983113564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:40:33.512212766 +0000 UTC m=+1508.037432358" watchObservedRunningTime="2026-10-31 03:40:33.528356463 +0000 UTC m=+1508.053576052"
Oct 31 03:40:33.931688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.923780    5098 patch_prober.go:29] interesting pod/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body=
Oct 31 03:40:33.931688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.923842    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused"
Oct 31 03:40:33.931688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:33.928013    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"]
Oct 31 03:40:34.153779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.153733    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:40:34.157018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.156986    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:40:34.315846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.314744    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:40:34.319339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.319305    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:40:34.320813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.320792    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:40:34.463995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.463952    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl" event={"ID":"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1","Type":"ContainerStarted","Data":"151530a266bfe3f8086209df8737c0e45a82312f9988a4017d595b6d082c19e3"}
Oct 31 03:40:34.660193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.660151    5098 scope.go:122] "RemoveContainer" containerID="727baa2294063c07cbd3b48abaa619d845062e19fb13e95d83e5f7e566646711"
Oct 31 03:40:34.924848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.924773    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:40:34.925955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:34.925931    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:40:35.475294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:35.475256    5098 generic.go:356] "Generic (PLEG): container finished" podID="d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" containerID="5e202ad9031899ace7a53e8089c64fe9d64cbd4820896b48e6defe1de352e26b" exitCode=0
Oct 31 03:40:35.475850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:35.475315    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl" event={"ID":"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1","Type":"ContainerDied","Data":"5e202ad9031899ace7a53e8089c64fe9d64cbd4820896b48e6defe1de352e26b"}
Oct 31 03:40:36.702847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.702812    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:36.807907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.807868    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\") pod \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") "
Oct 31 03:40:36.808170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.807996    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\") pod \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") "
Oct 31 03:40:36.808170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.808113    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\") pod \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\" (UID: \"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1\") "
Oct 31 03:40:36.808769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.808741    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca" pod "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" (UID: "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:36.809063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.809041    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:36.811253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.811201    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca" pod "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" (UID: "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:40:36.811727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.811549    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt" pod "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" (UID: "d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1"). InnerVolumeSpecName "kube-api-access-btrjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:36.911244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.911208    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btrjt\" (UniqueName: \"kubernetes.io/projected/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-kube-api-access-btrjt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:36.911244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:36.911242    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:37.496318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:37.496288    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"
Oct 31 03:40:37.496524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:37.496322    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl" event={"ID":"d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1","Type":"ContainerDied","Data":"151530a266bfe3f8086209df8737c0e45a82312f9988a4017d595b6d082c19e3"}
Oct 31 03:40:37.496524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:37.496376    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="151530a266bfe3f8086209df8737c0e45a82312f9988a4017d595b6d082c19e3"
Oct 31 03:40:37.678740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:37.678698    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:40:39.469841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:39.469811    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:40:41.390946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:41.390908    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:40:46.391837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:46.391765    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" podUID="ecc0acca-8037-4b84-9263-b32a193fd85a" containerName="payload-pre-processing" containerID="cri-o://f0fbbd0559787b5fb5315b3ea81609181f6df44fce5a808ed3ba810f91a048e6" gracePeriod=25
Oct 31 03:40:47.225335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.224065    5098 generic.go:356] "Generic (PLEG): container finished" podID="ecc0acca-8037-4b84-9263-b32a193fd85a" containerID="f0fbbd0559787b5fb5315b3ea81609181f6df44fce5a808ed3ba810f91a048e6" exitCode=0
Oct 31 03:40:47.225335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.224244    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" event={"ID":"ecc0acca-8037-4b84-9263-b32a193fd85a","Type":"ContainerDied","Data":"f0fbbd0559787b5fb5315b3ea81609181f6df44fce5a808ed3ba810f91a048e6"}
Oct 31 03:40:47.260690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.252858    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:40:47.290358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.290316    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\") pod \"ecc0acca-8037-4b84-9263-b32a193fd85a\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") "
Oct 31 03:40:47.324840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.297544    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume" pod "ecc0acca-8037-4b84-9263-b32a193fd85a" (UID: "ecc0acca-8037-4b84-9263-b32a193fd85a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:47.325559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.313997    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"]
Oct 31 03:40:47.325858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.325837    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8361df-dblln"]
Oct 31 03:40:47.345629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.339782    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\" (UniqueName: \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\") pod \"ecc0acca-8037-4b84-9263-b32a193fd85a\" (UID: \"ecc0acca-8037-4b84-9263-b32a193fd85a\") "
Oct 31 03:40:47.345629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.344698    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m" pod "ecc0acca-8037-4b84-9263-b32a193fd85a" (UID: "ecc0acca-8037-4b84-9263-b32a193fd85a"). InnerVolumeSpecName "kube-api-access-jwq6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:47.480756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.473330    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwq6m\" (UniqueName: \"kubernetes.io/projected/ecc0acca-8037-4b84-9263-b32a193fd85a-kube-api-access-jwq6m\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:47.480756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.473371    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc0acca-8037-4b84-9263-b32a193fd85a-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:47.614116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:47.614079    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="560fe01d-e6c2-4262-9017-b182b4fea1b7" path="/var/lib/kubelet/pods/560fe01d-e6c2-4262-9017-b182b4fea1b7/volumes"
Oct 31 03:40:48.234825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:48.234794    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"
Oct 31 03:40:48.235024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:48.234827    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67" event={"ID":"ecc0acca-8037-4b84-9263-b32a193fd85a","Type":"ContainerDied","Data":"5d5be4f1e23edec079ffccd4296035b2236d6bb9404f6dfa96570fc7751d8aaa"}
Oct 31 03:40:48.235024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:48.234886    5098 scope.go:122] "RemoveContainer" containerID="f0fbbd0559787b5fb5315b3ea81609181f6df44fce5a808ed3ba810f91a048e6"
Oct 31 03:40:48.281426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:48.281380    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:40:48.286128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:48.286102    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-74c760-db46879ff-6xz67"]
Oct 31 03:40:49.738035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:49.737994    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc0acca-8037-4b84-9263-b32a193fd85a" path="/var/lib/kubelet/pods/ecc0acca-8037-4b84-9263-b32a193fd85a/volumes"
Oct 31 03:40:50.749777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:50.749739    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:40:50.750197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:50.749806    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:40:51.226168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:51.226130    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:40:51.226347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:51.226325    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy" containerID="cri-o://2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26" gracePeriod=30
Oct 31 03:40:54.614013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:54.613975    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:40:54.625984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:54.614843    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy" containerID="cri-o://4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c" gracePeriod=30
Oct 31 03:40:55.424982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.424946    5098 generic.go:356] "Generic (PLEG): container finished" podID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerID="b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f" exitCode=0
Oct 31 03:40:55.425149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.425015    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerDied","Data":"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"}
Oct 31 03:40:55.425534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.425520    5098 scope.go:122] "RemoveContainer" containerID="b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"
Oct 31 03:40:55.427587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.427552    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:55.427746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.427693    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:55.427816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:55.427755    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:40:56.547225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.546863    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerStarted","Data":"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"}
Oct 31 03:40:56.551087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.550307    5098 generic.go:356] "Generic (PLEG): container finished" podID="63d5fdd6-21f3-4859-a497-8488c0163461" containerID="2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26" exitCode=0
Oct 31 03:40:56.551087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.550705    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" event={"ID":"63d5fdd6-21f3-4859-a497-8488c0163461","Type":"ContainerDied","Data":"2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26"}
Oct 31 03:40:56.721953 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:40:56.721902    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d5fdd6_21f3_4859_a497_8488c0163461.slice/crio-2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63d5fdd6_21f3_4859_a497_8488c0163461.slice/crio-conmon-2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:40:56.873542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.873512    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:40:56.879354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.879315    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:40:56.879687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.879652    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:40:56.941461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:56.941432    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:40:57.037866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.037829    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.037921    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.037970    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038060    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038129    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038136    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:57.038250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038151    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038333    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:57.038408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038380    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:40:57.038408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038395    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038441    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038516    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\") pod \"63d5fdd6-21f3-4859-a497-8488c0163461\" (UID: \"63d5fdd6-21f3-4859-a497-8488c0163461\") "
Oct 31 03:40:57.038570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038525    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:57.038777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.038626    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:57.039483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.039452    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.039483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.039485    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.039719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.039504    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.039719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.039519    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.039719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.039533    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63d5fdd6-21f3-4859-a497-8488c0163461-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.040695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.040660    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:40:57.040937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.040916    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:57.041134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.041115    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:40:57.041183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.041162    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9" pod "63d5fdd6-21f3-4859-a497-8488c0163461" (UID: "63d5fdd6-21f3-4859-a497-8488c0163461"). InnerVolumeSpecName "kube-api-access-h6lf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:40:57.142749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.141581    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63d5fdd6-21f3-4859-a497-8488c0163461-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.142749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.141644    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63d5fdd6-21f3-4859-a497-8488c0163461-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.142749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.141661    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6lf9\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-kube-api-access-h6lf9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.142749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.141676    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63d5fdd6-21f3-4859-a497-8488c0163461-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:40:57.585322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.585288    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"
Oct 31 03:40:57.585840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.585318    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv" event={"ID":"63d5fdd6-21f3-4859-a497-8488c0163461","Type":"ContainerDied","Data":"01780d88f3f3d3bd7ec532fc156213918e0025a7b2567da49835d92e8e04f027"}
Oct 31 03:40:57.585840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.585384    5098 scope.go:122] "RemoveContainer" containerID="2d676dd92c0bc75b18c87c415a442bd7db4ac4578d0d403db6dbdbb7d699cf26"
Oct 31 03:40:57.732479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.732357    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:40:57.733716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.733687    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-q6jmv"]
Oct 31 03:40:57.873690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.873043    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:40:57.873690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:57.873121    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:40:58.876594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:58.872885    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:40:58.876594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:58.872952    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:40:59.618510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:59.618464    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" path="/var/lib/kubelet/pods/63d5fdd6-21f3-4859-a497-8488c0163461/volumes"
Oct 31 03:40:59.876709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:59.875850    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:40:59.876709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:40:59.875915    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:41:00.259051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.258664    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436208    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436335    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436453    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436519    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436657    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436688    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436720    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436784    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.461723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.436854    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\") pod \"84e31255-64f6-419b-83a4-1a78222b96c8\" (UID: \"84e31255-64f6-419b-83a4-1a78222b96c8\") "
Oct 31 03:41:00.472261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.471850    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:00.472627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.472563    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.473797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.473750    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.477292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.477253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.497086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.496142    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.511953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.511917    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:00.512354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.512319    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:00.512495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.512448    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:41:00.512548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.512477    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf" pod "84e31255-64f6-419b-83a4-1a78222b96c8" (UID: "84e31255-64f6-419b-83a4-1a78222b96c8"). InnerVolumeSpecName "kube-api-access-j49xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:00.539298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.539263    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.539585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.539558    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.539785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.539767    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.539889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.539875    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/84e31255-64f6-419b-83a4-1a78222b96c8-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.539982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.539970    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.540065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.540053    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/84e31255-64f6-419b-83a4-1a78222b96c8-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.540145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.540134    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.540235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.540220    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j49xf\" (UniqueName: \"kubernetes.io/projected/84e31255-64f6-419b-83a4-1a78222b96c8-kube-api-access-j49xf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.540324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.540312    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/84e31255-64f6-419b-83a4-1a78222b96c8-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:00.616368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.616323    5098 generic.go:356] "Generic (PLEG): container finished" podID="84e31255-64f6-419b-83a4-1a78222b96c8" containerID="4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c" exitCode=0
Oct 31 03:41:00.616573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.616385    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"
Oct 31 03:41:00.616573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.616392    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerDied","Data":"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"}
Oct 31 03:41:00.616573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.616447    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5" event={"ID":"84e31255-64f6-419b-83a4-1a78222b96c8","Type":"ContainerDied","Data":"45700e678ad5b2eb13ecb3925ee0c09f957ee9beeec4eeec6c06a7bf1778b09b"}
Oct 31 03:41:00.616573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.616475    5098 scope.go:122] "RemoveContainer" containerID="4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"
Oct 31 03:41:00.635974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.635934    5098 scope.go:122] "RemoveContainer" containerID="db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"
Oct 31 03:41:00.685002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.684877    5098 scope.go:122] "RemoveContainer" containerID="4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"
Oct 31 03:41:00.686894 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:00.686854    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c\": container with ID starting with 4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c not found: ID does not exist" containerID="4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"
Oct 31 03:41:00.687101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.686908    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c"} err="failed to get container status \"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c\": rpc error: code = NotFound desc = could not find container \"4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c\": container with ID starting with 4c8a608072eccd97451aac9de3df6778ac21967e8e94f09f0ac35fbba594eb3c not found: ID does not exist"
Oct 31 03:41:00.687101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.686938    5098 scope.go:122] "RemoveContainer" containerID="db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"
Oct 31 03:41:00.687748 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:00.687696    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0\": container with ID starting with db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0 not found: ID does not exist" containerID="db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"
Oct 31 03:41:00.687748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.687736    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0"} err="failed to get container status \"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0\": rpc error: code = NotFound desc = could not find container \"db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0\": container with ID starting with db8a60860d27c2fa7c2f211c6944b012d42809f2e4e9b4a3b095f622f6108db0 not found: ID does not exist"
Oct 31 03:41:00.690663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.690595    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:41:00.696520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.696494    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-8361df-openshift-default-688c5b6559-9g7k5"]
Oct 31 03:41:00.874231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.874188    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:41:00.874428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:00.874264    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:41:01.183595 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:01.183508    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:41:01.295133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295099    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:41:01.295911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295894    5098 cpu_manager.go:414] "containerMap: removing container" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.295911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295912    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295932    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ecc0acca-8037-4b84-9263-b32a193fd85a" containerName="payload-pre-processing"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295938    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc0acca-8037-4b84-9263-b32a193fd85a" containerName="payload-pre-processing"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295955    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" containerName="revoke-keys"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295961    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" containerName="revoke-keys"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295986    5098 cpu_manager.go:414] "containerMap: removing container" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy"
Oct 31 03:41:01.296011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.295991    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy"
Oct 31 03:41:01.296235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.296113    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.296235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.296124    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc0acca-8037-4b84-9263-b32a193fd85a" containerName="payload-pre-processing"
Oct 31 03:41:01.296235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.296132    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" containerName="revoke-keys"
Oct 31 03:41:01.296235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.296140    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.296235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.296149    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="63d5fdd6-21f3-4859-a497-8488c0163461" containerName="istio-proxy"
Oct 31 03:41:01.301526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.301499    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.309735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.309703    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:41:01.310754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.310727    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5d9b6-dockercfg-bvnzp\""
Oct 31 03:41:01.310904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.310804    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5d9b6\""
Oct 31 03:41:01.383989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.383949    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:41:01.385038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.385008    5098 cpu_manager.go:414] "containerMap: removing container" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.385038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.385038    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" containerName="istio-proxy"
Oct 31 03:41:01.393092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.393060    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.449729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.448577    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.449729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.448770    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8dk2\" (UniqueName: \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.449729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.448808    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.449729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.448849    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpkd9\" (UniqueName: \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.585634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.585576    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:41:01.611646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.607851    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.611646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.608039    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8dk2\" (UniqueName: \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.611646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.608077    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.611646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.608115    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpkd9\" (UniqueName: \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.611646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.609550    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.664652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.628693    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.889313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.872860    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:41:01.889776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.889730    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:41:01.931185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.931143    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8dk2\" (UniqueName: \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\") pod \"payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:01.931752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.931725    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpkd9\" (UniqueName: \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\") pod \"payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:01.931909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:01.931732    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84e31255-64f6-419b-83a4-1a78222b96c8" path="/var/lib/kubelet/pods/84e31255-64f6-419b-83a4-1a78222b96c8/volumes"
Oct 31 03:41:02.030884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.030837    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:02.224944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.224892    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:02.437888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.437854    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:41:02.440767 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:02.440736    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b021c19_9ed3_4036_956f_5747a64b5e93.slice/crio-c9d39596925098b7409076c3af381adcc070c5c090d8aff80554f837b72d544b WatchSource:0}: Error finding container c9d39596925098b7409076c3af381adcc070c5c090d8aff80554f837b72d544b: Status 404 returned error can't find the container with id c9d39596925098b7409076c3af381adcc070c5c090d8aff80554f837b72d544b
Oct 31 03:41:02.729286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.729252    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:41:02.729561 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:02.729533    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9262e9b7_5033_4443_894d_ebbc0bab390e.slice/crio-2981fed050c350623e7ad8074a7ab3de1c0ec79aa63b19c2008feecfd29e4ca8 WatchSource:0}: Error finding container 2981fed050c350623e7ad8074a7ab3de1c0ec79aa63b19c2008feecfd29e4ca8: Status 404 returned error can't find the container with id 2981fed050c350623e7ad8074a7ab3de1c0ec79aa63b19c2008feecfd29e4ca8
Oct 31 03:41:02.800677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.800638    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" event={"ID":"9262e9b7-5033-4443-894d-ebbc0bab390e","Type":"ContainerStarted","Data":"2981fed050c350623e7ad8074a7ab3de1c0ec79aa63b19c2008feecfd29e4ca8"}
Oct 31 03:41:02.802621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.802573    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" event={"ID":"8b021c19-9ed3-4036-956f-5747a64b5e93","Type":"ContainerStarted","Data":"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"}
Oct 31 03:41:02.802813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.802681    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" event={"ID":"8b021c19-9ed3-4036-956f-5747a64b5e93","Type":"ContainerStarted","Data":"c9d39596925098b7409076c3af381adcc070c5c090d8aff80554f837b72d544b"}
Oct 31 03:41:02.802945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.802923    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:02.824001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.823936    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" podStartSLOduration=1.8239162279999999 podStartE2EDuration="1.823916228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:02.82237725 +0000 UTC m=+1537.347596851" watchObservedRunningTime="2026-10-31 03:41:02.823916228 +0000 UTC m=+1537.349135820"
Oct 31 03:41:02.878626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.876115    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused" start-of-body=
Oct 31 03:41:02.878626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.876179    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.183:15021/healthz/ready\": dial tcp 10.132.0.183:15021: connect: connection refused"
Oct 31 03:41:02.963019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:02.962980    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:41:03.022158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.022071    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.041626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.041579    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-73e222-serving-cert\""
Oct 31 03:41:03.055863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.051947    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:41:03.375972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.375928    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.376422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.376136    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.376422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.376169    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.376422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.376199    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-275lj\" (UniqueName: \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.478450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.478402    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.478691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.478473    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.478691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.478527    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-275lj\" (UniqueName: \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.478875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.478795    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.480242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.480205    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.482055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.482025    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.482387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.482363    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.490821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.490790    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-275lj\" (UniqueName: \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\") pod \"maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.588507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.588476    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:03.898626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.885202    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:41:03.898626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.887130    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:41:03.898626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.893904    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:41:03.922400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.921931    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" event={"ID":"9262e9b7-5033-4443-894d-ebbc0bab390e","Type":"ContainerStarted","Data":"e5ab75a1bbc975eb9af8d6b6653df9cd9e4effe94643540f09ebffde9e20716d"}
Oct 31 03:41:03.922400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.922260    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:03.961738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:03.961673    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" podStartSLOduration=2.961653009 podStartE2EDuration="2.961653009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:03.961148889 +0000 UTC m=+1538.486368479" watchObservedRunningTime="2026-10-31 03:41:03.961653009 +0000 UTC m=+1538.486872599"
Oct 31 03:41:04.238001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:04.237965    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:41:04.933250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:04.933144    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" event={"ID":"ad699c26-7f19-4bcb-8505-397e23a3ceb5","Type":"ContainerStarted","Data":"337cb3fb34e4848cdc7a89ad263d628aa38f162a51e2a5b21b5ebbd842336763"}
Oct 31 03:41:04.933250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:04.933215    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" event={"ID":"ad699c26-7f19-4bcb-8505-397e23a3ceb5","Type":"ContainerStarted","Data":"34df790dbba8cc54dc3492d55cc5bbf8365b207901a689e5c89aa250be69033f"}
Oct 31 03:41:04.954537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:04.954479    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" podStartSLOduration=2.533983838 podStartE2EDuration="2.954462689s" totalImagesPullingTime="420.478851ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:04.953090625 +0000 UTC m=+1539.478310231" watchObservedRunningTime="2026-10-31 03:41:04.954462689 +0000 UTC m=+1539.479682277"
Oct 31 03:41:05.943279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:05.943252    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:07.419632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:07.412266    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"]
Oct 31 03:41:07.857748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:07.857713    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-74c760-xswdl"]
Oct 31 03:41:09.275703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:09.275671    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"]
Oct 31 03:41:09.286133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:09.284493    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-619d46-7fxd9"]
Oct 31 03:41:09.663252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:09.650522    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df57478-c4d2-4c36-aad4-b3d43d3737a5" path="/var/lib/kubelet/pods/9df57478-c4d2-4c36-aad4-b3d43d3737a5/volumes"
Oct 31 03:41:09.663252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:09.651454    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1" path="/var/lib/kubelet/pods/d1e6845c-a15d-4a3b-a4e5-b7d72c8644a1/volumes"
Oct 31 03:41:12.032541 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:12.032507    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:41:13.375232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.375193    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"]
Oct 31 03:41:13.388012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.387981    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.393314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.393283    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"]
Oct 31 03:41:13.396486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.396459    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:41:13.398136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.398105    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:41:13.475637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.475561    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.475835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.475753    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.475901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.475889    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rd2s\" (UniqueName: \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.578263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.578222    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.578437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.578391    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.578573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.578559    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rd2s\" (UniqueName: \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.579088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.579065    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.580761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.580737    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.588444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.588410    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rd2s\" (UniqueName: \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\") pod \"maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.711360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.711257    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:13.926241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:13.926209    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:41:14.068342 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:14.068304    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb487599a_6c33_4c24_84f7_2640604a4abc.slice/crio-b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51 WatchSource:0}: Error finding container b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51: Status 404 returned error can't find the container with id b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51
Oct 31 03:41:14.069505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:14.069476    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"]
Oct 31 03:41:14.151771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:14.151730    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj" event={"ID":"b487599a-6c33-4c24-84f7-2640604a4abc","Type":"ContainerStarted","Data":"b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51"}
Oct 31 03:41:15.166300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:15.166253    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj" event={"ID":"b487599a-6c33-4c24-84f7-2640604a4abc","Type":"ContainerStarted","Data":"40575fab36b47c3dfdaf87d49d096bda97ef7ead529eaf739241248718674251"}
Oct 31 03:41:15.190663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:15.190589    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj" podStartSLOduration=2.190572177 podStartE2EDuration="2.190572177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:15.189077499 +0000 UTC m=+1549.714297087" watchObservedRunningTime="2026-10-31 03:41:15.190572177 +0000 UTC m=+1549.715791795"
Oct 31 03:41:15.947768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:15.947734    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:41:16.230554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.230456    5098 generic.go:356] "Generic (PLEG): container finished" podID="b487599a-6c33-4c24-84f7-2640604a4abc" containerID="40575fab36b47c3dfdaf87d49d096bda97ef7ead529eaf739241248718674251" exitCode=0
Oct 31 03:41:16.230554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.230550    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj" event={"ID":"b487599a-6c33-4c24-84f7-2640604a4abc","Type":"ContainerDied","Data":"40575fab36b47c3dfdaf87d49d096bda97ef7ead529eaf739241248718674251"}
Oct 31 03:41:16.482957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.482857    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:41:16.483204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.483152    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" containerID="cri-o://87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806" gracePeriod=30
Oct 31 03:41:16.977925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.977884    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"]
Oct 31 03:41:17.000592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:16.998975    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.003824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.003795    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"]
Oct 31 03:41:17.076762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.076700    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.076762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.076768    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-247r2\" (UniqueName: \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.077036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.076979    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.178048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.178007    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-247r2\" (UniqueName: \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.178233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.178176    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.178493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.178476    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.179115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.179094    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.180754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.180724    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.187450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.187419    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-247r2\" (UniqueName: \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\") pod \"maas-api-revoke-keys-e2e-api-b-a3b986-hh47r\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.332561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.332520    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:17.426145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.426112    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:17.484153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.483551    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\") pod \"b487599a-6c33-4c24-84f7-2640604a4abc\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") "
Oct 31 03:41:17.484153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.483703    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\" (UniqueName: \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\") pod \"b487599a-6c33-4c24-84f7-2640604a4abc\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") "
Oct 31 03:41:17.484153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.483751    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\") pod \"b487599a-6c33-4c24-84f7-2640604a4abc\" (UID: \"b487599a-6c33-4c24-84f7-2640604a4abc\") "
Oct 31 03:41:17.486252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.485850    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca" pod "b487599a-6c33-4c24-84f7-2640604a4abc" (UID: "b487599a-6c33-4c24-84f7-2640604a4abc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:17.488846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.488821    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca" pod "b487599a-6c33-4c24-84f7-2640604a4abc" (UID: "b487599a-6c33-4c24-84f7-2640604a4abc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:41:17.491677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.489101    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s" pod "b487599a-6c33-4c24-84f7-2640604a4abc" (UID: "b487599a-6c33-4c24-84f7-2640604a4abc"). InnerVolumeSpecName "kube-api-access-8rd2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:17.600628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.600424    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:17.600628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.600462    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rd2s\" (UniqueName: \"kubernetes.io/projected/b487599a-6c33-4c24-84f7-2640604a4abc-kube-api-access-8rd2s\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:17.600628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.600481    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b487599a-6c33-4c24-84f7-2640604a4abc-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:17.658332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:17.658304    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"]
Oct 31 03:41:17.660366 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:17.660334    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b796bca_de98_4951_accd_452910ee4f5a.slice/crio-b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799 WatchSource:0}: Error finding container b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799: Status 404 returned error can't find the container with id b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799
Oct 31 03:41:18.250810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.250774    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r" event={"ID":"3b796bca-de98-4951-accd-452910ee4f5a","Type":"ContainerStarted","Data":"64b26c6e38d9b58d4a5350545165e1905a893c690556dc63f1409682ecadad5f"}
Oct 31 03:41:18.251029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.250831    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r" event={"ID":"3b796bca-de98-4951-accd-452910ee4f5a","Type":"ContainerStarted","Data":"b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799"}
Oct 31 03:41:18.252626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.252583    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj" event={"ID":"b487599a-6c33-4c24-84f7-2640604a4abc","Type":"ContainerDied","Data":"b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51"}
Oct 31 03:41:18.252763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.252641    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1ab5baaea5e48991618aabbfefcc3be24cff6be3240d5a18e1cf58c30df0c51"
Oct 31 03:41:18.252763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.252592    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"
Oct 31 03:41:18.277583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.277529    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r" podStartSLOduration=2.27751428 podStartE2EDuration="2.27751428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:18.27325838 +0000 UTC m=+1552.798477969" watchObservedRunningTime="2026-10-31 03:41:18.27751428 +0000 UTC m=+1552.802733915"
Oct 31 03:41:18.888846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.888811    5098 patch_prober.go:29] interesting pod/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:41:18.889247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:18.888869    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:41:19.263888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:19.263848    5098 generic.go:356] "Generic (PLEG): container finished" podID="3b796bca-de98-4951-accd-452910ee4f5a" containerID="64b26c6e38d9b58d4a5350545165e1905a893c690556dc63f1409682ecadad5f" exitCode=0
Oct 31 03:41:19.264111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:19.263932    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r" event={"ID":"3b796bca-de98-4951-accd-452910ee4f5a","Type":"ContainerDied","Data":"64b26c6e38d9b58d4a5350545165e1905a893c690556dc63f1409682ecadad5f"}
Oct 31 03:41:20.001235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.001199    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:41:20.001726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.001422    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy" containerID="cri-o://e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31" gracePeriod=30
Oct 31 03:41:20.208280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.208241    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"]
Oct 31 03:41:20.250117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.250075    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39805b5d-l5g9x"]
Oct 31 03:41:20.609078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.609001    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:20.656563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.656529    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\") pod \"3b796bca-de98-4951-accd-452910ee4f5a\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") "
Oct 31 03:41:20.656802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.656598    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\" (UniqueName: \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\") pod \"3b796bca-de98-4951-accd-452910ee4f5a\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") "
Oct 31 03:41:20.656802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.656784    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\") pod \"3b796bca-de98-4951-accd-452910ee4f5a\" (UID: \"3b796bca-de98-4951-accd-452910ee4f5a\") "
Oct 31 03:41:20.657186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.657158    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca" pod "3b796bca-de98-4951-accd-452910ee4f5a" (UID: "3b796bca-de98-4951-accd-452910ee4f5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:20.657693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.657671    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:20.659024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.659000    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca" pod "3b796bca-de98-4951-accd-452910ee4f5a" (UID: "3b796bca-de98-4951-accd-452910ee4f5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:41:20.659138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.659116    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2" pod "3b796bca-de98-4951-accd-452910ee4f5a" (UID: "3b796bca-de98-4951-accd-452910ee4f5a"). InnerVolumeSpecName "kube-api-access-247r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:20.759894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.759856    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b796bca-de98-4951-accd-452910ee4f5a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:20.759894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:20.759891    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-247r2\" (UniqueName: \"kubernetes.io/projected/3b796bca-de98-4951-accd-452910ee4f5a-kube-api-access-247r2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:21.342715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:21.342679    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"
Oct 31 03:41:21.342715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:21.342698    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r" event={"ID":"3b796bca-de98-4951-accd-452910ee4f5a","Type":"ContainerDied","Data":"b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799"}
Oct 31 03:41:21.343171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:21.342751    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4f54cd7c6307ba70a44e13fb99e6494083a6245dceddaeeb6f0fc128da71799"
Oct 31 03:41:21.616190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:21.616156    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a738cf03-feb0-459e-af9c-7cc303d6672d" path="/var/lib/kubelet/pods/a738cf03-feb0-459e-af9c-7cc303d6672d/volumes"
Oct 31 03:41:22.321532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.321506    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:41:22.354099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.354068    5098 generic.go:356] "Generic (PLEG): container finished" podID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerID="87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806" exitCode=0
Oct 31 03:41:22.354536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.354115    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"
Oct 31 03:41:22.354536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.354162    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerDied","Data":"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"}
Oct 31 03:41:22.354536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.354223    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl" event={"ID":"97e84386-1465-4921-85b6-78d9d0edd9ec","Type":"ContainerDied","Data":"9b62b09377c9cbf3333b7b140d19dc09b5c79fdb016bea610ce105f51cfe4a72"}
Oct 31 03:41:22.354536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.354247    5098 scope.go:122] "RemoveContainer" containerID="87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"
Oct 31 03:41:22.365810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.365789    5098 scope.go:122] "RemoveContainer" containerID="b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"
Oct 31 03:41:22.380439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.380409    5098 scope.go:122] "RemoveContainer" containerID="87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"
Oct 31 03:41:22.381093 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:22.381067    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806\": container with ID starting with 87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806 not found: ID does not exist" containerID="87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"
Oct 31 03:41:22.381207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.381101    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806"} err="failed to get container status \"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806\": rpc error: code = NotFound desc = could not find container \"87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806\": container with ID starting with 87304592d885f971d15f7d69f73f63aa735839497556112156b2f69d4c536806 not found: ID does not exist"
Oct 31 03:41:22.381207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.381122    5098 scope.go:122] "RemoveContainer" containerID="b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"
Oct 31 03:41:22.381442 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:22.381423    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f\": container with ID starting with b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f not found: ID does not exist" containerID="b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"
Oct 31 03:41:22.381486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.381448    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f"} err="failed to get container status \"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f\": rpc error: code = NotFound desc = could not find container \"b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f\": container with ID starting with b25fccaee4d148f4c714b0a71c0ee630af0aa52400c0bb258974eb902a2b5d0f not found: ID does not exist"
Oct 31 03:41:22.396993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.396963    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397117    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397187    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397320    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:22.397456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397361    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397488    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397509    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:22.397672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397558    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397584    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:22.397672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397600    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397684    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397716    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\") pod \"97e84386-1465-4921-85b6-78d9d0edd9ec\" (UID: \"97e84386-1465-4921-85b6-78d9d0edd9ec\") "
Oct 31 03:41:22.397931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397836    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:22.398067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.397973    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:22.398503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.398479    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.398656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.398510    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97e84386-1465-4921-85b6-78d9d0edd9ec-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.398656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.398527    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.398656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.398540    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.398656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.398553    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.402062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.402033    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:41:22.402169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.402126    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:22.402169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.402141    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "kube-api-access-lln2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:22.402357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.402338    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token" pod "97e84386-1465-4921-85b6-78d9d0edd9ec" (UID: "97e84386-1465-4921-85b6-78d9d0edd9ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:22.500437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.500404    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.500437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.500431    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.500437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.500442    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97e84386-1465-4921-85b6-78d9d0edd9ec-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.500696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.500452    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lln2p\" (UniqueName: \"kubernetes.io/projected/97e84386-1465-4921-85b6-78d9d0edd9ec-kube-api-access-lln2p\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:22.702772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.702736    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:41:22.712536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:22.712498    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-74c760-openshift-default-7f6b5f579-hkpfl"]
Oct 31 03:41:23.614285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:23.614250    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" path="/var/lib/kubelet/pods/97e84386-1465-4921-85b6-78d9d0edd9ec/volumes"
Oct 31 03:41:23.736502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:23.736468    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:41:23.754562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:23.752508    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:41:25.314040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.314009    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:41:25.432435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432392    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.432721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432505    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.432721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432542    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.432721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432577    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.432721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432657    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.432721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432693    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.433038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432906    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.433038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.432919    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:25.433038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433000    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.433200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433131    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\") pod \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\" (UID: \"c1fa7fd6-c485-42ff-9680-8d76dcad89bb\") "
Oct 31 03:41:25.433200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433140    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:25.433315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433230    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:25.433414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433383    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:25.433675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.433601    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:25.434145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.434118    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.434277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.434151    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.434277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.434165    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.434277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.434180    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.434277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.434195    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.435426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.435398    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:41:25.436039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.436006    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:25.436190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.436093    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:41:25.436284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.436198    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb" pod "c1fa7fd6-c485-42ff-9680-8d76dcad89bb" (UID: "c1fa7fd6-c485-42ff-9680-8d76dcad89bb"). InnerVolumeSpecName "kube-api-access-dgxqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:25.536258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.536225    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.536258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.536256    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.536569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.536266    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgxqb\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-kube-api-access-dgxqb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.536569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.536276    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1fa7fd6-c485-42ff-9680-8d76dcad89bb-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:25.541985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.541950    5098 generic.go:356] "Generic (PLEG): container finished" podID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerID="e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31" exitCode=0
Oct 31 03:41:25.542126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.542036    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"
Oct 31 03:41:25.542126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.542033    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerDied","Data":"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"}
Oct 31 03:41:25.542203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.542163    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54" event={"ID":"c1fa7fd6-c485-42ff-9680-8d76dcad89bb","Type":"ContainerDied","Data":"8828b052bf81e28a89e8b432aef599e1905e73a99a63a6ed43136b6da1f03096"}
Oct 31 03:41:25.542203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.542191    5098 scope.go:122] "RemoveContainer" containerID="e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"
Oct 31 03:41:25.555808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.555784    5098 scope.go:122] "RemoveContainer" containerID="82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"
Oct 31 03:41:25.567755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.567718    5098 scope.go:122] "RemoveContainer" containerID="e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"
Oct 31 03:41:25.568078 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:25.568055    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31\": container with ID starting with e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31 not found: ID does not exist" containerID="e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"
Oct 31 03:41:25.568137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.568087    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31"} err="failed to get container status \"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31\": rpc error: code = NotFound desc = could not find container \"e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31\": container with ID starting with e6f6a3126ece2fbcc7590bd78a5a5dbff4dab8a797f13bbbc86ff0591d27fd31 not found: ID does not exist"
Oct 31 03:41:25.568137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.568107    5098 scope.go:122] "RemoveContainer" containerID="82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"
Oct 31 03:41:25.568392 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:25.568365    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3\": container with ID starting with 82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3 not found: ID does not exist" containerID="82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"
Oct 31 03:41:25.568508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.568429    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3"} err="failed to get container status \"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3\": rpc error: code = NotFound desc = could not find container \"82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3\": container with ID starting with 82ac803ecb5a7ac8d67c6f93cbf85c34201f943ea362e6761d9218fab7f8e6d3 not found: ID does not exist"
Oct 31 03:41:25.575495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.575465    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:41:25.581531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.581505    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-619d46-openshift-default-65646c4d47-f8x54"]
Oct 31 03:41:25.615440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:25.615409    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" path="/var/lib/kubelet/pods/c1fa7fd6-c485-42ff-9680-8d76dcad89bb/volumes"
Oct 31 03:41:27.587025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.586992    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:41:27.587922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.587900    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b487599a-6c33-4c24-84f7-2640604a4abc" containerName="revoke-keys"
Oct 31 03:41:27.587922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.587922    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b487599a-6c33-4c24-84f7-2640604a4abc" containerName="revoke-keys"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.587980    5098 cpu_manager.go:414] "containerMap: removing container" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.587985    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.587994    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588002    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588016    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588022    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588030    5098 cpu_manager.go:414] "containerMap: removing container" podUID="3b796bca-de98-4951-accd-452910ee4f5a" containerName="revoke-keys"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588034    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b796bca-de98-4951-accd-452910ee4f5a" containerName="revoke-keys"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588048    5098 cpu_manager.go:414] "containerMap: removing container" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.588055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588052    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.588492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588210    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.588492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588222    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588235    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fa7fd6-c485-42ff-9680-8d76dcad89bb" containerName="istio-proxy"
Oct 31 03:41:27.588492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588243    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b796bca-de98-4951-accd-452910ee4f5a" containerName="revoke-keys"
Oct 31 03:41:27.588492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.588251    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b487599a-6c33-4c24-84f7-2640604a4abc" containerName="revoke-keys"
Oct 31 03:41:27.593995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.593965    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.603781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.603665    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a5d9b6-serving-cert\""
Oct 31 03:41:27.638750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.638710    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:41:27.660488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.660441    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:41:27.661556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.661528    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="97e84386-1465-4921-85b6-78d9d0edd9ec" containerName="istio-proxy"
Oct 31 03:41:27.665593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.665569    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:27.668805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.668780    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.668959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.668843    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6vlg\" (UniqueName: \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.669696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.669210    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.669696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.669384    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.701392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.701335    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:41:27.702523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.702492    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-034fe8\""
Oct 31 03:41:27.703655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.703631    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-034fe8-dockercfg-4xvqp\""
Oct 31 03:41:27.728032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.727994    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:41:27.734486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.734458    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:27.753165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.752822    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:41:27.771045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771011    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.771200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771086    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6vlg\" (UniqueName: \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.771200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771133    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:27.771455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771435    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.771593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771576    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.771739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771625    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:27.771739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771664    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ch8x\" (UniqueName: \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:27.771857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771745    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.771857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.771771    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtz4\" (UniqueName: \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:27.774108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.774084    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.774244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.774225    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:27.878724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.878422    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:27.878724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.878489    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ch8x\" (UniqueName: \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:27.878724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.878546    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtz4\" (UniqueName: \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:27.878724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.878711    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:27.918629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.881669    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:28.001704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:27.999433    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:28.010661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.010402    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtz4\" (UniqueName: \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\") pod \"payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:28.039518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.039482    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ch8x\" (UniqueName: \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\") pod \"payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") " pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:28.071630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.062382    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6vlg\" (UniqueName: \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\") pod \"maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:28.075631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.072891    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:28.237227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.236915    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:28.287237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.286908    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:28.502373 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:28.502328    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49cb135a_57ad_4060_8719_3341d1a152d7.slice/crio-1d5c3a2a8eff3225cd3d5f6a4407772ff44eb588b08c3b09c416e034c66a0a2f WatchSource:0}: Error finding container 1d5c3a2a8eff3225cd3d5f6a4407772ff44eb588b08c3b09c416e034c66a0a2f: Status 404 returned error can't find the container with id 1d5c3a2a8eff3225cd3d5f6a4407772ff44eb588b08c3b09c416e034c66a0a2f
Oct 31 03:41:28.506046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.506016    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:41:28.574661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.574577    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" event={"ID":"49cb135a-57ad-4060-8719-3341d1a152d7","Type":"ContainerStarted","Data":"1d5c3a2a8eff3225cd3d5f6a4407772ff44eb588b08c3b09c416e034c66a0a2f"}
Oct 31 03:41:28.600899 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:28.600865    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5c3968_8757_492b_88c5_f01a4c8db0dc.slice/crio-62d9306739c27b1dc88195a8bf0b9975c5bea08cf2b694fb84485f8467220bf5 WatchSource:0}: Error finding container 62d9306739c27b1dc88195a8bf0b9975c5bea08cf2b694fb84485f8467220bf5: Status 404 returned error can't find the container with id 62d9306739c27b1dc88195a8bf0b9975c5bea08cf2b694fb84485f8467220bf5
Oct 31 03:41:28.606726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.605881    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:41:28.627719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.627681    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:41:28.628499 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:28.628470    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9afa47_8e4b_492e_9088_c12b2dd52cdf.slice/crio-d72ebdecea8f37e9006952833a87ce6ecf18caf536a660071150a0fd5fd5ec1f WatchSource:0}: Error finding container d72ebdecea8f37e9006952833a87ce6ecf18caf536a660071150a0fd5fd5ec1f: Status 404 returned error can't find the container with id d72ebdecea8f37e9006952833a87ce6ecf18caf536a660071150a0fd5fd5ec1f
Oct 31 03:41:28.737985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.737850    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" podUID="39710305-ec38-4c9a-89e4-e8d0b8652414" containerName="payload-processing" containerID="cri-o://cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b" gracePeriod=25
Oct 31 03:41:28.753809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:28.753738    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" podUID="bf2aa246-0acb-4379-b3f4-4f091c4db938" containerName="payload-pre-processing" containerID="cri-o://3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7" gracePeriod=25
Oct 31 03:41:29.334879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.334796    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:41:29.406130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.406090    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:41:29.493749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.493563    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\") pod \"bf2aa246-0acb-4379-b3f4-4f091c4db938\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") "
Oct 31 03:41:29.498657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.498620    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\" (UniqueName: \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\") pod \"39710305-ec38-4c9a-89e4-e8d0b8652414\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") "
Oct 31 03:41:29.499011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.498987    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\") pod \"39710305-ec38-4c9a-89e4-e8d0b8652414\" (UID: \"39710305-ec38-4c9a-89e4-e8d0b8652414\") "
Oct 31 03:41:29.511305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.508708    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\" (UniqueName: \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\") pod \"bf2aa246-0acb-4379-b3f4-4f091c4db938\" (UID: \"bf2aa246-0acb-4379-b3f4-4f091c4db938\") "
Oct 31 03:41:29.514190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.510284    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume" pod "bf2aa246-0acb-4379-b3f4-4f091c4db938" (UID: "bf2aa246-0acb-4379-b3f4-4f091c4db938"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:29.514190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.514148    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf2aa246-0acb-4379-b3f4-4f091c4db938-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:29.529359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.526884    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume" pod "39710305-ec38-4c9a-89e4-e8d0b8652414" (UID: "39710305-ec38-4c9a-89e4-e8d0b8652414"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:41:29.563787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.563380    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz" pod "39710305-ec38-4c9a-89e4-e8d0b8652414" (UID: "39710305-ec38-4c9a-89e4-e8d0b8652414"). InnerVolumeSpecName "kube-api-access-m4txz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:29.563787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.563529    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444" pod "bf2aa246-0acb-4379-b3f4-4f091c4db938" (UID: "bf2aa246-0acb-4379-b3f4-4f091c4db938"). InnerVolumeSpecName "kube-api-access-l2444". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:41:29.612096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.611980    5098 generic.go:356] "Generic (PLEG): container finished" podID="39710305-ec38-4c9a-89e4-e8d0b8652414" containerID="cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b" exitCode=0
Oct 31 03:41:29.613031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.613002    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"
Oct 31 03:41:29.615890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.615811    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39710305-ec38-4c9a-89e4-e8d0b8652414-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:29.615890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.615843    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2444\" (UniqueName: \"kubernetes.io/projected/bf2aa246-0acb-4379-b3f4-4f091c4db938-kube-api-access-l2444\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:29.615890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.615857    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4txz\" (UniqueName: \"kubernetes.io/projected/39710305-ec38-4c9a-89e4-e8d0b8652414-kube-api-access-m4txz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:41:29.618643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.618559    5098 generic.go:356] "Generic (PLEG): container finished" podID="bf2aa246-0acb-4379-b3f4-4f091c4db938" containerID="3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7" exitCode=0
Oct 31 03:41:29.620720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.618959    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"
Oct 31 03:41:29.644811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.644776    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" event={"ID":"39710305-ec38-4c9a-89e4-e8d0b8652414","Type":"ContainerDied","Data":"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"}
Oct 31 03:41:29.647084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.647056    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:29.647290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.647279    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:29.647511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.647076    5098 scope.go:122] "RemoveContainer" containerID="cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"
Oct 31 03:41:29.648237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.648223    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:29.648619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.648575    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp" event={"ID":"39710305-ec38-4c9a-89e4-e8d0b8652414","Type":"ContainerDied","Data":"895597c915193c92c53fbdc3ed35b3427f239459dafb6ef22bcf0d70e5d16fe7"}
Oct 31 03:41:29.648801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.648783    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" event={"ID":"bf2aa246-0acb-4379-b3f4-4f091c4db938","Type":"ContainerDied","Data":"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"}
Oct 31 03:41:29.648897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.648887    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp" event={"ID":"bf2aa246-0acb-4379-b3f4-4f091c4db938","Type":"ContainerDied","Data":"2b42973e8372b6f0be28175a09a6a7968c9d8a80222c9efb31fe3ff8f3639c9b"}
Oct 31 03:41:29.648966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.648955    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" event={"ID":"49cb135a-57ad-4060-8719-3341d1a152d7","Type":"ContainerStarted","Data":"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"}
Oct 31 03:41:29.649035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.649026    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" event={"ID":"9e5c3968-8757-492b-88c5-f01a4c8db0dc","Type":"ContainerStarted","Data":"3d7abdfd3b2a2bb9bf5554f800efe76bc513cfdd2939f31a93d37fb54e22451e"}
Oct 31 03:41:29.649157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.649143    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" event={"ID":"9e5c3968-8757-492b-88c5-f01a4c8db0dc","Type":"ContainerStarted","Data":"62d9306739c27b1dc88195a8bf0b9975c5bea08cf2b694fb84485f8467220bf5"}
Oct 31 03:41:29.649284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.649272    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" event={"ID":"2a9afa47-8e4b-492e-9088-c12b2dd52cdf","Type":"ContainerStarted","Data":"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"}
Oct 31 03:41:29.649488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.649432    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" event={"ID":"2a9afa47-8e4b-492e-9088-c12b2dd52cdf","Type":"ContainerStarted","Data":"d72ebdecea8f37e9006952833a87ce6ecf18caf536a660071150a0fd5fd5ec1f"}
Oct 31 03:41:29.664239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.664203    5098 scope.go:122] "RemoveContainer" containerID="cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"
Oct 31 03:41:29.664830 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:29.664798    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b\": container with ID starting with cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b not found: ID does not exist" containerID="cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"
Oct 31 03:41:29.664982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.664843    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b"} err="failed to get container status \"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b\": rpc error: code = NotFound desc = could not find container \"cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b\": container with ID starting with cda03bf02d642feda52ea86ec653a1e18b8f97ce2987c5121189ff4346f5594b not found: ID does not exist"
Oct 31 03:41:29.664982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.664868    5098 scope.go:122] "RemoveContainer" containerID="3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"
Oct 31 03:41:29.668488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.668428    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" podStartSLOduration=2.6684084649999997 podStartE2EDuration="2.668408465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:29.661331064 +0000 UTC m=+1564.186550652" watchObservedRunningTime="2026-10-31 03:41:29.668408465 +0000 UTC m=+1564.193628055"
Oct 31 03:41:29.682043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.682020    5098 scope.go:122] "RemoveContainer" containerID="3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"
Oct 31 03:41:29.682412 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:29.682388    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7\": container with ID starting with 3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7 not found: ID does not exist" containerID="3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"
Oct 31 03:41:29.682485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.682421    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7"} err="failed to get container status \"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7\": rpc error: code = NotFound desc = could not find container \"3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7\": container with ID starting with 3b36067b5d8bd3e91e4524d4c995a61de381d24da0c3b6ab5bdecef6d9a3f6e7 not found: ID does not exist"
Oct 31 03:41:29.695981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.695923    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" podStartSLOduration=2.695910263 podStartE2EDuration="2.695910263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:29.693643177 +0000 UTC m=+1564.218862770" watchObservedRunningTime="2026-10-31 03:41:29.695910263 +0000 UTC m=+1564.221129868"
Oct 31 03:41:29.720437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.720401    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:41:29.728539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.728500    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-34dbeb-64df696775-tdkjp"]
Oct 31 03:41:29.787851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.787786    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" podStartSLOduration=2.270842604 podStartE2EDuration="2.787765114s" totalImagesPullingTime="516.92251ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:29.78472119 +0000 UTC m=+1564.309940779" watchObservedRunningTime="2026-10-31 03:41:29.787765114 +0000 UTC m=+1564.312984705"
Oct 31 03:41:29.914358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.913424    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:41:29.997727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:29.997699    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-34dbeb-5b66c749f-q4gfp"]
Oct 31 03:41:31.065626 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:31.065569    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:41:31.614314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:31.614277    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39710305-ec38-4c9a-89e4-e8d0b8652414" path="/var/lib/kubelet/pods/39710305-ec38-4c9a-89e4-e8d0b8652414/volumes"
Oct 31 03:41:31.614839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:31.614823    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf2aa246-0acb-4379-b3f4-4f091c4db938" path="/var/lib/kubelet/pods/bf2aa246-0acb-4379-b3f4-4f091c4db938/volumes"
Oct 31 03:41:34.712140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:34.712103    5098 scope.go:122] "RemoveContainer" containerID="5117566ee5e6290696927c2a8678f63172a0f330e81f06625822173453a093fe"
Oct 31 03:41:34.723314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:34.723294    5098 scope.go:122] "RemoveContainer" containerID="ad6819c150dcdac8f628fcf4c46708b8a400219b56d67a1164891d420f3fd716"
Oct 31 03:41:34.734477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:34.734455    5098 scope.go:122] "RemoveContainer" containerID="33bdfc42b4563dc842ba78e66fdb118c5a2251f4af77280293a22d643d3fe33e"
Oct 31 03:41:36.493417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:36.493375    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:41:36.693630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:36.693497    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:41:41.700870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:41.700832    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:41:41.702232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:41.702204    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:41:49.593369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.593329    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:41:49.594185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594167    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bf2aa246-0acb-4379-b3f4-4f091c4db938" containerName="payload-pre-processing"
Oct 31 03:41:49.594228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594186    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf2aa246-0acb-4379-b3f4-4f091c4db938" containerName="payload-pre-processing"
Oct 31 03:41:49.594228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594214    5098 cpu_manager.go:414] "containerMap: removing container" podUID="39710305-ec38-4c9a-89e4-e8d0b8652414" containerName="payload-processing"
Oct 31 03:41:49.594228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594219    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39710305-ec38-4c9a-89e4-e8d0b8652414" containerName="payload-processing"
Oct 31 03:41:49.594364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594354    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf2aa246-0acb-4379-b3f4-4f091c4db938" containerName="payload-pre-processing"
Oct 31 03:41:49.594404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.594370    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="39710305-ec38-4c9a-89e4-e8d0b8652414" containerName="payload-processing"
Oct 31 03:41:49.598340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.598319    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.604714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.604685    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-034fe8-serving-cert\""
Oct 31 03:41:49.623456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.623421    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:41:49.648367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.648327    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.648594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.648489    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.648695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.648673    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgbpp\" (UniqueName: \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.651737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.648978    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.753536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.753486    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.753772 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:49.753663    5098 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-serving-cert: secret "maas-api-e2e-inf-a-034fe8-serving-cert" not found
Oct 31 03:41:49.753772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.753672    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.753772 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:41:49.753746    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls podName:d8bb65e9-ed5b-4549-870a-55f5f45d77e3 nodeName:}" failed. No retries permitted until 2026-10-31 03:41:50.253729232 +0000 UTC m=+1584.778948802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls") pod "maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" (UID: "d8bb65e9-ed5b-4549-870a-55f5f45d77e3") : secret "maas-api-e2e-inf-a-034fe8-serving-cert" not found
Oct 31 03:41:49.753965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.753885    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.754024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.753998    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgbpp\" (UniqueName: \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.754650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.754626    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.756281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.756250    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:49.771630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:49.765837    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgbpp\" (UniqueName: \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:50.282413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:50.282375    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:50.285093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:50.285063    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") pod \"maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:50.526564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:50.526519    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:50.935648 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:41:50.935357    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bb65e9_ed5b_4549_870a_55f5f45d77e3.slice/crio-a11ece6365eee6a1316895547d69f7a5bd4b43c4a780aae49ef8fe28d5790b0a WatchSource:0}: Error finding container a11ece6365eee6a1316895547d69f7a5bd4b43c4a780aae49ef8fe28d5790b0a: Status 404 returned error can't find the container with id a11ece6365eee6a1316895547d69f7a5bd4b43c4a780aae49ef8fe28d5790b0a
Oct 31 03:41:50.937004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:50.936976    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:41:50.953397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:50.953354    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" event={"ID":"d8bb65e9-ed5b-4549-870a-55f5f45d77e3","Type":"ContainerStarted","Data":"a11ece6365eee6a1316895547d69f7a5bd4b43c4a780aae49ef8fe28d5790b0a"}
Oct 31 03:41:51.965588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:51.965552    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" event={"ID":"d8bb65e9-ed5b-4549-870a-55f5f45d77e3","Type":"ContainerStarted","Data":"ee1cf95d51466468288d9c5c87a840dbe05243c6312839de67f58d1086ad8907"}
Oct 31 03:41:51.966035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:51.965650    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:41:51.985633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:51.985565    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" podStartSLOduration=2.583550694 podStartE2EDuration="2.985550773s" totalImagesPullingTime="402.000079ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:41:51.983871422 +0000 UTC m=+1586.509091013" watchObservedRunningTime="2026-10-31 03:41:51.985550773 +0000 UTC m=+1586.510770408"
Oct 31 03:41:55.739559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:55.739521    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"]
Oct 31 03:41:55.744363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:55.744329    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-34dbeb-9lzpj"]
Oct 31 03:41:57.661026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:57.660962    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b487599a-6c33-4c24-84f7-2640604a4abc" path="/var/lib/kubelet/pods/b487599a-6c33-4c24-84f7-2640604a4abc/volumes"
Oct 31 03:41:57.980944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:41:57.980866    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:42:01.064123 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:01.064084    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:42:02.289706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.289648    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:42:02.300056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.300024    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.306894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.306855    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:42:02.318102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.318058    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-86a40d\""
Oct 31 03:42:02.380800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.380766    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-86a40d-dockercfg-qbgt9\""
Oct 31 03:42:02.489632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.471992    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzhhf\" (UniqueName: \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.600688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.490268    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.707422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.707371    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.762195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.762152    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzhhf\" (UniqueName: \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.767277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.736422    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.793641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.788776    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzhhf\" (UniqueName: \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\") pod \"payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:02.922883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:02.922800    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:03.469595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:03.469548    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:42:03.475375 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:42:03.475340    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d0ca89e_3ab1_42cc_a513_f76920562b57.slice/crio-e44b5fb5e80cbb0d8ca7c6ea3ae68e31ab0073bf3b8bd5981bf5f0c56c484262 WatchSource:0}: Error finding container e44b5fb5e80cbb0d8ca7c6ea3ae68e31ab0073bf3b8bd5981bf5f0c56c484262: Status 404 returned error can't find the container with id e44b5fb5e80cbb0d8ca7c6ea3ae68e31ab0073bf3b8bd5981bf5f0c56c484262
Oct 31 03:42:04.247352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:04.247315    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" event={"ID":"7d0ca89e-3ab1-42cc-a513-f76920562b57","Type":"ContainerStarted","Data":"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"}
Oct 31 03:42:04.247573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:04.247381    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" event={"ID":"7d0ca89e-3ab1-42cc-a513-f76920562b57","Type":"ContainerStarted","Data":"e44b5fb5e80cbb0d8ca7c6ea3ae68e31ab0073bf3b8bd5981bf5f0c56c484262"}
Oct 31 03:42:04.247573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:04.247410    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:04.267657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:04.267565    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" podStartSLOduration=2.26754349 podStartE2EDuration="2.26754349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:04.264123391 +0000 UTC m=+1598.789342981" watchObservedRunningTime="2026-10-31 03:42:04.26754349 +0000 UTC m=+1598.792763082"
Oct 31 03:42:05.181268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.181232    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"]
Oct 31 03:42:05.186991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.186928    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.192542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.192492    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:42:05.192542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.192493    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:42:05.193886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.193864    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"]
Oct 31 03:42:05.277324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.277286    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.277510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.277447    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npmrm\" (UniqueName: \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.277732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.277713    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.379187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.379148    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npmrm\" (UniqueName: \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.379397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.379374    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.379482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.379465    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.380186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.380157    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.381876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.381852    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.388813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.388786    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npmrm\" (UniqueName: \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\") pod \"maas-api-revoke-keys-e2e-api-b-007d82-lvwnz\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.516881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.516775    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:05.829696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:05.829655    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"]
Oct 31 03:42:05.829925 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:42:05.829721    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680fecc7_1d4e_4b70_b55b_daa32e496569.slice/crio-bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06 WatchSource:0}: Error finding container bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06: Status 404 returned error can't find the container with id bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06
Oct 31 03:42:06.267557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:06.267525    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz" event={"ID":"680fecc7-1d4e-4b70-b55b-daa32e496569","Type":"ContainerStarted","Data":"bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06"}
Oct 31 03:42:07.278067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:07.278030    5098 generic.go:356] "Generic (PLEG): container finished" podID="680fecc7-1d4e-4b70-b55b-daa32e496569" containerID="2b8b9f39e2f54d3211eec0a65733c1c371d7bf20722af3941996ee2de934a2f8" exitCode=0
Oct 31 03:42:07.278483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:07.278076    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz" event={"ID":"680fecc7-1d4e-4b70-b55b-daa32e496569","Type":"ContainerDied","Data":"2b8b9f39e2f54d3211eec0a65733c1c371d7bf20722af3941996ee2de934a2f8"}
Oct 31 03:42:08.479498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.479467    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:08.522375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.522342    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\" (UniqueName: \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\") pod \"680fecc7-1d4e-4b70-b55b-daa32e496569\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") "
Oct 31 03:42:08.522547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.522427    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\") pod \"680fecc7-1d4e-4b70-b55b-daa32e496569\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") "
Oct 31 03:42:08.522623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.522561    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\") pod \"680fecc7-1d4e-4b70-b55b-daa32e496569\" (UID: \"680fecc7-1d4e-4b70-b55b-daa32e496569\") "
Oct 31 03:42:08.522978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.522942    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca" pod "680fecc7-1d4e-4b70-b55b-daa32e496569" (UID: "680fecc7-1d4e-4b70-b55b-daa32e496569"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:08.523462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.523436    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:08.524856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.524831    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm" pod "680fecc7-1d4e-4b70-b55b-daa32e496569" (UID: "680fecc7-1d4e-4b70-b55b-daa32e496569"). InnerVolumeSpecName "kube-api-access-npmrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:08.525265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.525250    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca" pod "680fecc7-1d4e-4b70-b55b-daa32e496569" (UID: "680fecc7-1d4e-4b70-b55b-daa32e496569"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:42:08.625135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.625039    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npmrm\" (UniqueName: \"kubernetes.io/projected/680fecc7-1d4e-4b70-b55b-daa32e496569-kube-api-access-npmrm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:08.625135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:08.625082    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/680fecc7-1d4e-4b70-b55b-daa32e496569-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:09.299060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:09.299022    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"
Oct 31 03:42:09.299255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:09.299056    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz" event={"ID":"680fecc7-1d4e-4b70-b55b-daa32e496569","Type":"ContainerDied","Data":"bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06"}
Oct 31 03:42:09.299255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:09.299111    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfb3f26e75d0e8d2c9943c5384c709711a8a7f1aa7a52de1afffd56c881e7a06"
Oct 31 03:42:11.396431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:11.396401    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"]
Oct 31 03:42:11.404009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:11.403973    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3b986-hh47r"]
Oct 31 03:42:11.625958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:11.625919    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b796bca-de98-4951-accd-452910ee4f5a" path="/var/lib/kubelet/pods/3b796bca-de98-4951-accd-452910ee4f5a/volumes"
Oct 31 03:42:15.286631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:15.285836    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:42:19.412011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:19.409592    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:42:19.412011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:19.410362    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy" containerID="cri-o://b237f2136ca1ba0fa537a49af6541d292fcf4e3da904134d00b6106d46301373" gracePeriod=30
Oct 31 03:42:22.896529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.896488    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:42:22.897657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.897597    5098 cpu_manager.go:414] "containerMap: removing container" podUID="680fecc7-1d4e-4b70-b55b-daa32e496569" containerName="revoke-keys"
Oct 31 03:42:22.897657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.897641    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="680fecc7-1d4e-4b70-b55b-daa32e496569" containerName="revoke-keys"
Oct 31 03:42:22.897917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.897901    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="680fecc7-1d4e-4b70-b55b-daa32e496569" containerName="revoke-keys"
Oct 31 03:42:22.902036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.902012    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:22.909036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.908992    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-46ebfe\""
Oct 31 03:42:22.917284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.917248    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:42:22.983885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.983849    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djvsg\" (UniqueName: \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:22.984179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:22.984143    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.128631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.121996    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djvsg\" (UniqueName: \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.128631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.122229    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.128631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.123472    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.336681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.336640    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djvsg\" (UniqueName: \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\") pod \"payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.524124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.524083    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:23.920302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:23.920245    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:42:24.642253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.642153    5098 generic.go:356] "Generic (PLEG): container finished" podID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerID="b237f2136ca1ba0fa537a49af6541d292fcf4e3da904134d00b6106d46301373" exitCode=0
Oct 31 03:42:24.642752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.642250    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerDied","Data":"b237f2136ca1ba0fa537a49af6541d292fcf4e3da904134d00b6106d46301373"}
Oct 31 03:42:24.643399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.643379    5098 scope.go:122] "RemoveContainer" containerID="1a849f443ed87d1e4dca9fba4ab6af5f2df1e1f5cd016c041c2016b9a5b8109b"
Oct 31 03:42:24.647402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.647374    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" event={"ID":"cd96482e-2160-438b-a8e0-7ce09f950cf3","Type":"ContainerStarted","Data":"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"}
Oct 31 03:42:24.647545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.647423    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" event={"ID":"cd96482e-2160-438b-a8e0-7ce09f950cf3","Type":"ContainerStarted","Data":"b9c74272c066973a43d461aeb10625d5ed289ec41034568a0fbf3530210a0ced"}
Oct 31 03:42:24.647813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.647794    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:24.684769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.684132    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" podStartSLOduration=2.684109746 podStartE2EDuration="2.684109746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:24.666598456 +0000 UTC m=+1619.191818046" watchObservedRunningTime="2026-10-31 03:42:24.684109746 +0000 UTC m=+1619.209329338"
Oct 31 03:42:24.808475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.808437    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:42:24.812491 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812463    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.812706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812597    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.812779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812702    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.812779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812742    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.812902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812885    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.812961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812886    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:42:24.812961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812932    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.813079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.812968    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.813079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813063    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.813191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813148    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\") pod \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\" (UID: \"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15\") "
Oct 31 03:42:24.813246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813191    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:24.813864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813378    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:24.813864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813507    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:24.813864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.813587    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:24.814130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.814053    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.814130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.814076    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.814130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.814092    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.814130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.814106    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.814130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.814119    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.815656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.815573    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:42:24.815795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.815751    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:42:24.815863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.815817    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "kube-api-access-8dqpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:24.816702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.816677    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token" pod "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" (UID: "4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:42:24.916023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.915976    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dqpj\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-kube-api-access-8dqpj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.916023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.916022    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.916254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.916038    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:24.916254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:24.916052    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:42:25.696583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:25.696550    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"
Oct 31 03:42:25.699950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:25.699750    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l" event={"ID":"4ca6a4d2-214c-4e46-8faf-c00c6ca0df15","Type":"ContainerDied","Data":"6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae"}
Oct 31 03:42:25.700113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:25.699981    5098 scope.go:122] "RemoveContainer" containerID="b237f2136ca1ba0fa537a49af6541d292fcf4e3da904134d00b6106d46301373"
Oct 31 03:42:25.729432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:25.729399    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:42:25.734071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:25.734047    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a3b986-openshift-default-d4978ddd9-fcr2l"]
Oct 31 03:42:26.249296 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:26.249257    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:42:27.281775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.281737    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:42:27.282647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.282615    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.282647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.282633    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.283208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.282770    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.283208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.282782    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.287814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.287795    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.297902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.297866    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-50b5f2\""
Oct 31 03:42:27.298048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.297923    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-50b5f2-dockercfg-5pgbs\""
Oct 31 03:42:27.299046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.299025    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:42:27.308192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.308161    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:42:27.309461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.309435    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.309522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.309465    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" containerName="istio-proxy"
Oct 31 03:42:27.315213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.315188    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.333998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.333967    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:42:27.393167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.393131    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8fkj\" (UniqueName: \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.393360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.393338    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.393403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.393371    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9d7j\" (UniqueName: \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.393585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.393552    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.525629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.520400    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8fkj\" (UniqueName: \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.525629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.520711    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.525629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.520745    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9d7j\" (UniqueName: \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.525629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.520904    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.534671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.526586    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.556638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.549690    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8fkj\" (UniqueName: \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\") pod \"payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.561780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.557456    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.601632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.601349    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9d7j\" (UniqueName: \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\") pod \"payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") " pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.608125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.608094    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:27.640180 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.638886    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:27.850922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:27.846286    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" path="/var/lib/kubelet/pods/4ca6a4d2-214c-4e46-8faf-c00c6ca0df15/volumes"
Oct 31 03:42:28.148997 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:42:28.148961    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d0f2052_ee90_4e30_9c88_81227ca3a5b8.slice/crio-e6d8d038fce0748d48d32a851e43dc39b9daaeafab566baf10ccaf4682dfb687 WatchSource:0}: Error finding container e6d8d038fce0748d48d32a851e43dc39b9daaeafab566baf10ccaf4682dfb687: Status 404 returned error can't find the container with id e6d8d038fce0748d48d32a851e43dc39b9daaeafab566baf10ccaf4682dfb687
Oct 31 03:42:28.149677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.149536    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:42:28.176142 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:42:28.176108    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd442e0ea_dd06_4492_bf3b_6d65627ecd87.slice/crio-2e44c90e4357d0cfe32b28d1ca52603f6a1551fd75c8a77f2c83722c78883f54 WatchSource:0}: Error finding container 2e44c90e4357d0cfe32b28d1ca52603f6a1551fd75c8a77f2c83722c78883f54: Status 404 returned error can't find the container with id 2e44c90e4357d0cfe32b28d1ca52603f6a1551fd75c8a77f2c83722c78883f54
Oct 31 03:42:28.177509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.177486    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.940616    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" event={"ID":"0d0f2052-ee90-4e30-9c88-81227ca3a5b8","Type":"ContainerStarted","Data":"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"}
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.940687    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" event={"ID":"0d0f2052-ee90-4e30-9c88-81227ca3a5b8","Type":"ContainerStarted","Data":"e6d8d038fce0748d48d32a851e43dc39b9daaeafab566baf10ccaf4682dfb687"}
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.945448    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.948562    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" event={"ID":"d442e0ea-dd06-4492-bf3b-6d65627ecd87","Type":"ContainerStarted","Data":"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"}
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.948662    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" event={"ID":"d442e0ea-dd06-4492-bf3b-6d65627ecd87","Type":"ContainerStarted","Data":"2e44c90e4357d0cfe32b28d1ca52603f6a1551fd75c8a77f2c83722c78883f54"}
Oct 31 03:42:28.972243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.950269    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:28.985478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:28.985426    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" podStartSLOduration=1.985409809 podStartE2EDuration="1.985409809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:28.977955427 +0000 UTC m=+1623.503175018" watchObservedRunningTime="2026-10-31 03:42:28.985409809 +0000 UTC m=+1623.510629398"
Oct 31 03:42:29.002621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:29.002563    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" podStartSLOduration=2.002550278 podStartE2EDuration="2.002550278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:42:29.000514862 +0000 UTC m=+1623.525734477" watchObservedRunningTime="2026-10-31 03:42:29.002550278 +0000 UTC m=+1623.527769866"
Oct 31 03:42:30.279243 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:30.279211    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:42:31.060800 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:31.060756    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:42:34.835850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:34.835819    5098 scope.go:122] "RemoveContainer" containerID="488923b302ab3f0ff338af97c0617412137af562af9497fee136458809db1ba6"
Oct 31 03:42:34.848781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:34.848750    5098 scope.go:122] "RemoveContainer" containerID="de274da5932ef9c36fdd98ea374e12f0d3d9e58cc66221b9968774ea4f44360b"
Oct 31 03:42:35.770362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:35.763913    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:42:40.001628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:39.998829    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:42:40.400927 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:40.400888    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache]"
Oct 31 03:42:40.971362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:40.971330    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:42:41.155715 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:41.155675    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:42:42.028351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:42.028314    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"]
Oct 31 03:42:42.034639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:42.034588    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50b44448-rfgpj"]
Oct 31 03:42:43.722400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:43.722357    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac" path="/var/lib/kubelet/pods/1c4a7a1a-3e83-493b-b8dc-5b0c72bb98ac/volumes"
Oct 31 03:42:50.531524 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:50.531477    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:42:55.707700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:55.707650    5098 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod4ca6a4d2-214c-4e46-8faf-c00c6ca0df15"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod4ca6a4d2-214c-4e46-8faf-c00c6ca0df15] : Timed out while waiting for systemd to remove kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice"
Oct 31 03:42:55.707700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:42:55.707697    5098 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="4ca6a4d2-214c-4e46-8faf-c00c6ca0df15" err="failed to delete cgroup paths for [kubepods burstable pod4ca6a4d2-214c-4e46-8faf-c00c6ca0df15] : unable to destroy cgroup paths for cgroup [kubepods burstable pod4ca6a4d2-214c-4e46-8faf-c00c6ca0df15] : Timed out while waiting for systemd to remove kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice"
Oct 31 03:42:56.332480 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:42:56.332444    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:00.550793 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:00.550752    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:01.060392 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:01.060346    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:43:10.656982 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:10.656896    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:11.154527 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:11.154481    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:13.670754 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:13.670707    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:13.671320 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:13.671290    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:15.961503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:15.961460    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"]
Oct 31 03:43:15.976144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:15.976084    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-007d82-lvwnz"]
Oct 31 03:43:17.614986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:17.614952    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="680fecc7-1d4e-4b70-b55b-daa32e496569" path="/var/lib/kubelet/pods/680fecc7-1d4e-4b70-b55b-daa32e496569/volumes"
Oct 31 03:43:20.835684 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:20.835640    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice/crio-6409fb85b4db7dc9992d5bada75b9637afef3013958fb298de721fd272714aae\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca6a4d2_214c_4e46_8faf_c00c6ca0df15.slice\": RecentStats: unable to find data in memory cache]"
Oct 31 03:43:24.235950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:24.235908    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:43:24.236666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:24.236181    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy" containerID="cri-o://d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5" gracePeriod=30
Oct 31 03:43:25.722645 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:25.722594    5098 fsHandler.go:119] failed to collect filesystem stats - rootDiskErr: could not stat "/var/lib/containers/storage/overlay/896ee00b02dad5a5b3c0de7fc04cd8e2f015080d7febe80311a0fe7bb96c576b/diff" to get inode usage: stat /var/lib/containers/storage/overlay/896ee00b02dad5a5b3c0de7fc04cd8e2f015080d7febe80311a0fe7bb96c576b/diff: no such file or directory, extraDiskErr: <nil>
Oct 31 03:43:29.554292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.554262    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:43:29.595985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.595946    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:43:29.597311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597281    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.597311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597310    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.597495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597336    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.597495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597345    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.597686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597668    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.597736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.597693    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerName="istio-proxy"
Oct 31 03:43:29.602490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.602465    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.619150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.619087    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-034fe8-openshift-default-dockercfg-vbg8p\""
Oct 31 03:43:29.701420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.701358    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.687456    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737135    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737176    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737254    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737337    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737364    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737480    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737531    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.737595    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\") pod \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\" (UID: \"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13\") "
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738030    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738309    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738343    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phs72\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738499    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738563    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738643    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.738775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738725    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.739602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738790    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.739602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.738985    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.739969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.739752    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:29.746772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.746733    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:29.747343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.747313    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:29.747597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.747532    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:29.749894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.749864    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:29.750266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.750243    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:43:29.750523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.750503    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:43:29.757385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.757326    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "kube-api-access-9tn8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:43:29.758198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.758019    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo" pod "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" (UID: "2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:43:29.840526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.840484    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.840734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.840707    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.840824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.840759    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.840886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.840855    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phs72\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.840998    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841046    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841116    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841212    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841253    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841361    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841555    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841677    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841698    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841714    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tn8j\" (UniqueName: \"kubernetes.io/projected/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-kube-api-access-9tn8j\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841722    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841733    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841748    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.841767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841767    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.842087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841781    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.842087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841796    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:43:29.842087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841846    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.842087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.841921    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.843552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.843529    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.843852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.843832    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.855717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.855685    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.855876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.855792    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phs72\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:29.937537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.937499    5098 generic.go:356] "Generic (PLEG): container finished" podID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" containerID="d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5" exitCode=0
Oct 31 03:43:29.937748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.937558    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"
Oct 31 03:43:29.937748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.937580    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerDied","Data":"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"}
Oct 31 03:43:29.937748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.937654    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm" event={"ID":"2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13","Type":"ContainerDied","Data":"292a5bbab6ec62d445ac5830f3bb3bdc7232ba664c3d2e1ef36972943b730675"}
Oct 31 03:43:29.937748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.937681    5098 scope.go:122] "RemoveContainer" containerID="d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"
Oct 31 03:43:29.958343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.958315    5098 scope.go:122] "RemoveContainer" containerID="93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"
Oct 31 03:43:29.971668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.971600    5098 scope.go:122] "RemoveContainer" containerID="d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"
Oct 31 03:43:29.972029 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:29.972005    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5\": container with ID starting with d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5 not found: ID does not exist" containerID="d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"
Oct 31 03:43:29.972136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.972041    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5"} err="failed to get container status \"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5\": rpc error: code = NotFound desc = could not find container \"d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5\": container with ID starting with d5c307288f00e0058bc9c6b844ac3c2edf9dac499c9e83ede4ddb25b1e0383c5 not found: ID does not exist"
Oct 31 03:43:29.972136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.972067    5098 scope.go:122] "RemoveContainer" containerID="93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"
Oct 31 03:43:29.972236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.972217    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:43:29.972370 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:29.972339    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2\": container with ID starting with 93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2 not found: ID does not exist" containerID="93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"
Oct 31 03:43:29.972409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.972381    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2"} err="failed to get container status \"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2\": rpc error: code = NotFound desc = could not find container \"93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2\": container with ID starting with 93d6f6472ef93c25cb31127b948944c8e7283980eba9a1791d89bea665d29ee2 not found: ID does not exist"
Oct 31 03:43:29.977070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:29.977049    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-007d82-openshift-default-66f8dc9685-7ksgm"]
Oct 31 03:43:30.010676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.010643    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:30.159339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.159294    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:43:30.454977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.452829    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:43:30.529794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.529711    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:30.530258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.530224    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:30.530446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.530424    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:43:30.960681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.960639    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" event={"ID":"13e29d8b-1433-449a-b73c-c1431fcd4ca5","Type":"ContainerStarted","Data":"cf6caa107bac2bca7daa7597aca9d7d5330ed6b36e6af720f8918f50018d3e81"}
Oct 31 03:43:30.963122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:30.963091    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" event={"ID":"13e29d8b-1433-449a-b73c-c1431fcd4ca5","Type":"ContainerStarted","Data":"4d08887c128f274bc25899c48d90af0b83da4c0e367ac2df74b3d522af0307b4"}
Oct 31 03:43:31.011161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:31.011130    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:31.011759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:31.011700    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podStartSLOduration=25.011685382 podStartE2EDuration="25.011685382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:43:31.002461656 +0000 UTC m=+1685.527681244" watchObservedRunningTime="2026-10-31 03:43:31.011685382 +0000 UTC m=+1685.536905046"
Oct 31 03:43:31.013089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:31.013063    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:31.013173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:31.013115    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:31.063351 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:43:31.063311    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:43:31.618704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:31.618665    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13" path="/var/lib/kubelet/pods/2acab3bb-8ff4-4413-8f3a-c6a44d8f8b13/volumes"
Oct 31 03:43:32.012690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:32.011991    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:32.012690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:32.012068    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:33.011765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:33.011717    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:33.011958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:33.011793    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:34.012044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:34.012003    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:34.012576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:34.012061    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:34.893566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:34.893533    5098 scope.go:122] "RemoveContainer" containerID="b0696479ad5bf2cb3855915692a48f211c8e3da257686341591827eb57f7237a"
Oct 31 03:43:34.906382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:34.906356    5098 scope.go:122] "RemoveContainer" containerID="6132d2dfe3f70c7a98bbb48a62ad0f30a873040e4f1392a5f71354880bd54dd3"
Oct 31 03:43:35.012060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:35.012022    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:35.012480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:35.012080    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:36.012094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:36.012051    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:36.012534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:36.012122    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:37.011152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:37.011116    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body=
Oct 31 03:43:37.011374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:37.011182    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused"
Oct 31 03:43:38.204628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:38.197195    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:38.204628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:38.198178    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:43:38.254513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:43:38.247705    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:44:01.059807 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:44:01.059763    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:44:06.822501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:06.821529    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:44:06.822501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:06.821924    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy" containerID="cri-o://5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3" gracePeriod=30
Oct 31 03:44:12.169902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.169874    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:44:12.207709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.207674    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:44:12.208662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.208640    5098 cpu_manager.go:414] "containerMap: removing container" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy"
Oct 31 03:44:12.208735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.208663    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy"
Oct 31 03:44:12.208839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.208827    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerName="istio-proxy"
Oct 31 03:44:12.213068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.213030    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.223165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.223117    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-46ebfe-openshift-default-dockercfg-zlk2v\""
Oct 31 03:44:12.227272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.227233    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:44:12.329264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329226    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.329464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329337    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.329464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329410    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.329562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.329645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329572    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.329706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329690    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329800    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329700    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329884    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329920    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.329919    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330010    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\") pod \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\" (UID: \"963eff1f-4fe4-4d79-ba3b-7ecd695e356c\") "
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330117    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330128    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330381    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330554    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330575    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k576z\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.330681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330622    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330800    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330863    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.330976    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.331003    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.331189    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332013    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332039    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332057    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332074    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.332172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332088    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.332707 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332683    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:12.332792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.332768    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:44:12.333116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.333096    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:12.333287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.333267    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx" pod "963eff1f-4fe4-4d79-ba3b-7ecd695e356c" (UID: "963eff1f-4fe4-4d79-ba3b-7ecd695e356c"). InnerVolumeSpecName "kube-api-access-skggx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.469948    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470011    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470122    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470267    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470447    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470532    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k576z\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.470583    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471008    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471082    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471297    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471320    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skggx\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-kube-api-access-skggx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471337    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.471682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.471359    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/963eff1f-4fe4-4d79-ba3b-7ecd695e356c-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:12.483636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.474839    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.483636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.475242    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.483636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.475662    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.483636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.480269    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.483636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.480978    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.547655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.530486    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k576z\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.555271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.555230    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.557747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.557707    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.564483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.564451    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\") pod \"e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") " pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:12.775695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.772574    5098 generic.go:356] "Generic (PLEG): container finished" podID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" containerID="5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3" exitCode=0
Oct 31 03:44:12.775695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.772866    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" event={"ID":"963eff1f-4fe4-4d79-ba3b-7ecd695e356c","Type":"ContainerDied","Data":"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"}
Oct 31 03:44:12.775695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.772931    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm" event={"ID":"963eff1f-4fe4-4d79-ba3b-7ecd695e356c","Type":"ContainerDied","Data":"0449de88a1590c66888990f37b767ab308781a1c0c6e0aa39e50dd11d7e616dd"}
Oct 31 03:44:12.775695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.772961    5098 scope.go:122] "RemoveContainer" containerID="5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"
Oct 31 03:44:12.775695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.773187    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"
Oct 31 03:44:12.798105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.798078    5098 scope.go:122] "RemoveContainer" containerID="5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"
Oct 31 03:44:12.798534 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:44:12.798499    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3\": container with ID starting with 5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3 not found: ID does not exist" containerID="5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"
Oct 31 03:44:12.798725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.798548    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3"} err="failed to get container status \"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3\": rpc error: code = NotFound desc = could not find container \"5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3\": container with ID starting with 5d9b8821244b3ce7779b66d37b373eec208eef63a05ffa7685267d718649edc3 not found: ID does not exist"
Oct 31 03:44:12.807238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.807193    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:44:12.815348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.815319    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-tmjmm"]
Oct 31 03:44:12.845360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:12.845322    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:13.262580 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:44:13.262539    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1d86cb_9463_42fd_b6eb_3360dbe651f5.slice/crio-6d608aa294d75515c3795d1b2285cd3b2b38dee67760577086c8ed8f3c0d5d5f WatchSource:0}: Error finding container 6d608aa294d75515c3795d1b2285cd3b2b38dee67760577086c8ed8f3c0d5d5f: Status 404 returned error can't find the container with id 6d608aa294d75515c3795d1b2285cd3b2b38dee67760577086c8ed8f3c0d5d5f
Oct 31 03:44:13.263424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.263329    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:44:13.267460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.267412    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:13.267576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.267492    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:13.267576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.267525    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:13.617183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.617105    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="963eff1f-4fe4-4d79-ba3b-7ecd695e356c" path="/var/lib/kubelet/pods/963eff1f-4fe4-4d79-ba3b-7ecd695e356c/volumes"
Oct 31 03:44:13.790462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.790419    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" event={"ID":"da1d86cb-9463-42fd-b6eb-3360dbe651f5","Type":"ContainerStarted","Data":"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"}
Oct 31 03:44:13.790731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.790477    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" event={"ID":"da1d86cb-9463-42fd-b6eb-3360dbe651f5","Type":"ContainerStarted","Data":"6d608aa294d75515c3795d1b2285cd3b2b38dee67760577086c8ed8f3c0d5d5f"}
Oct 31 03:44:13.819938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.819859    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podStartSLOduration=25.819840234 podStartE2EDuration="25.819840234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:13.815012628 +0000 UTC m=+1728.340232217" watchObservedRunningTime="2026-10-31 03:44:13.819840234 +0000 UTC m=+1728.345059824"
Oct 31 03:44:13.845532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:13.845493    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:14.890862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:14.887991    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:14.890862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:14.888056    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:15.846682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:15.846639    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:15.846901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:15.846704    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:16.620738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.620700    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:44:16.627782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.627737    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.629662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.629062    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:44:16.642421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.642388    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b68780\""
Oct 31 03:44:16.642572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.642546    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b68780-dockercfg-4cn5v\""
Oct 31 03:44:16.738896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.738848    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk256\" (UniqueName: \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.739632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.739196    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.860453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.860403    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk256\" (UniqueName: \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.860721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.860698    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.861807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.861774    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:16.862220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.862191    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:16.862313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:16.862246    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:17.157631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:17.156558    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk256\" (UniqueName: \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\") pod \"payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:17.278889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:17.265006    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:17.720682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:17.720660    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:44:17.846328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:17.846287    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:17.846509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:17.846354    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:18.187114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.187020    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" event={"ID":"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5","Type":"ContainerStarted","Data":"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"}
Oct 31 03:44:18.187114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.187076    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" event={"ID":"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5","Type":"ContainerStarted","Data":"ad3cc1c592f493bc52e100df129a0b515097cade8419bf462a4d55305c8555a9"}
Oct 31 03:44:18.187317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.187235    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:18.207597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.207540    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" podStartSLOduration=2.207524229 podStartE2EDuration="2.207524229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:18.205884431 +0000 UTC m=+1732.731104025" watchObservedRunningTime="2026-10-31 03:44:18.207524229 +0000 UTC m=+1732.732743817"
Oct 31 03:44:18.846632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.845763    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:18.846632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:18.845825    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:19.904630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:19.887709    5098 patch_prober.go:29] interesting pod/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body=
Oct 31 03:44:19.904630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:19.887801    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused"
Oct 31 03:44:20.932390 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:20.932298    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:20.932963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:20.932883    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:20.933936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:20.933911    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:44:22.158669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:22.158630    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:44:22.159144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:22.158858    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy" containerID="cri-o://95ba22e14ac52439506c263760930b9fea6b4c5d823ba286aaac5630a21e1e25" gracePeriod=30
Oct 31 03:44:27.339502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.339469    5098 generic.go:356] "Generic (PLEG): container finished" podID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerID="95ba22e14ac52439506c263760930b9fea6b4c5d823ba286aaac5630a21e1e25" exitCode=0
Oct 31 03:44:27.339913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.339541    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" event={"ID":"05eaf985-a4d8-4b80-9432-8397e07f3061","Type":"ContainerDied","Data":"95ba22e14ac52439506c263760930b9fea6b4c5d823ba286aaac5630a21e1e25"}
Oct 31 03:44:27.497726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.497692    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:44:27.528406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.528366    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.528617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.528482    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.528617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.528523    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.528617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.528581    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.530301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.528672    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.530301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.530116    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.530767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.530350    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.530767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.530407    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.530767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.530579    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\") pod \"05eaf985-a4d8-4b80-9432-8397e07f3061\" (UID: \"05eaf985-a4d8-4b80-9432-8397e07f3061\") "
Oct 31 03:44:27.531933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.531893    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:44:27.533196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.533168    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:27.533368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.533287    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:27.533459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.533389    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:27.533624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.533583    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:27.533856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.533831    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:44:27.534330 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.534302    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:27.535144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.535119    5098 cpu_manager.go:414] "containerMap: removing container" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy"
Oct 31 03:44:27.535232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.535145    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy"
Oct 31 03:44:27.535323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.535302    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "kube-api-access-wp9rs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:44:27.535378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.535332    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" containerName="istio-proxy"
Oct 31 03:44:27.535602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.535580    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:44:27.540039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.539998    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo" pod "05eaf985-a4d8-4b80-9432-8397e07f3061" (UID: "05eaf985-a4d8-4b80-9432-8397e07f3061"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:44:27.542889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.541095    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.548161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.547831    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-86a40d-openshift-default-dockercfg-49mvt\""
Oct 31 03:44:27.553702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.553672    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:44:27.632863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.632825    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4v6k\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.633766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.633693    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.634096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.634033    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.664998    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.670934    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.670978    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671162    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671264    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671429    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671702    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp9rs\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-kube-api-access-wp9rs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671723    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671748    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671765    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671782    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671796    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671812    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671877    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05eaf985-a4d8-4b80-9432-8397e07f3061-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.672629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.671900    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05eaf985-a4d8-4b80-9432-8397e07f3061-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:44:27.773951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.773858    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.773965    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774066    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4v6k\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774087    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774125    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774266    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774337    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774391    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774417    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774492    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774644    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.774936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774911    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.775033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.774971    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.775126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.775106    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.776483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.776454    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.776828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.776810    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.794488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.794440    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.794488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.794458    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4v6k\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:27.894125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:27.894092    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:28.189199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.188864    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:44:28.195130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.195083    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:28.195278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.195180    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:28.195278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.195231    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:44:28.352900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.352861    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" event={"ID":"46b30cc3-de1e-4da8-90f5-01e43bb4ca55","Type":"ContainerStarted","Data":"466caa37a368efe74f63f992f94aa665dfa44864b0c019893b6213656432c24d"}
Oct 31 03:44:28.353388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.352928    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" event={"ID":"46b30cc3-de1e-4da8-90f5-01e43bb4ca55","Type":"ContainerStarted","Data":"4fee762b542952c2c7543ebe07f3dc38a5571ed2eb76c6e2ba9c018f818e2d08"}
Oct 31 03:44:28.354855 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.354832    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"
Oct 31 03:44:28.354978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.354870    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw" event={"ID":"05eaf985-a4d8-4b80-9432-8397e07f3061","Type":"ContainerDied","Data":"f63b94dddd29e888d985dd0bf8e7f838efc11434f49e6e22d72abc365af08f7d"}
Oct 31 03:44:28.354978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.354943    5098 scope.go:122] "RemoveContainer" containerID="95ba22e14ac52439506c263760930b9fea6b4c5d823ba286aaac5630a21e1e25"
Oct 31 03:44:28.375859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.375799    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podStartSLOduration=1.3757794429999999 podStartE2EDuration="1.375779443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:44:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:44:28.37456554 +0000 UTC m=+1742.899785129" watchObservedRunningTime="2026-10-31 03:44:28.375779443 +0000 UTC m=+1742.900999033"
Oct 31 03:44:28.429656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.429286    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:44:28.470910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.470820    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-z7bhw"]
Oct 31 03:44:28.895130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:28.895098    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:29.244684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:29.237930    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:44:29.613651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:29.613601    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05eaf985-a4d8-4b80-9432-8397e07f3061" path="/var/lib/kubelet/pods/05eaf985-a4d8-4b80-9432-8397e07f3061/volumes"
Oct 31 03:44:29.918200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:29.915232    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 03:44:29.918200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:29.915309    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 03:44:30.894945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:30.894908    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 03:44:30.895334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:30.894968    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 03:44:31.087841 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:44:31.087804    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:44:31.895090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:31.895058    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 03:44:31.895594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:31.895124    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 03:44:32.894838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:32.894792    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 03:44:32.895038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:32.894855    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 03:44:33.894754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:33.894718    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body=
Oct 31 03:44:33.895196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:33.894784    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused"
Oct 31 03:44:34.993163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:34.993072    5098 scope.go:122] "RemoveContainer" containerID="3ccb0d276adc657a63567b3c149a7ab8b2f22fafad8652fb5c7eaaf6e00decbd"
Oct 31 03:44:35.008119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:35.008091    5098 scope.go:122] "RemoveContainer" containerID="e08d461462a97aaac66e2e4e66d3d23acde399e4696a782d167ac461a85725cf"
Oct 31 03:44:35.076909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:35.076689    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:35.077791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:35.077761    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:44:35.079064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:44:35.079014    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:45:00.154293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.154250    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:00.160729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.160697    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.172532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.172367    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:00.198686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.180960    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:45:00.296694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.296652    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls6wh\" (UniqueName: \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.296934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.296731    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.415736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.413901    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls6wh\" (UniqueName: \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.415736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.413994    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.415736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.415515    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.481920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.478554    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls6wh\" (UniqueName: \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.487584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.487206    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:00.782571 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:45:00.782528    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e061a0a_a744_4f1a_bade_0342abeadccc.slice/crio-8b4f2ac67e358b7eeaa7e6127fe3af137f5f5ac15647d0bf369ba5bdfa0f6444 WatchSource:0}: Error finding container 8b4f2ac67e358b7eeaa7e6127fe3af137f5f5ac15647d0bf369ba5bdfa0f6444: Status 404 returned error can't find the container with id 8b4f2ac67e358b7eeaa7e6127fe3af137f5f5ac15647d0bf369ba5bdfa0f6444
Oct 31 03:45:00.782739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.782697    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:00.887392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:00.887353    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerStarted","Data":"8b4f2ac67e358b7eeaa7e6127fe3af137f5f5ac15647d0bf369ba5bdfa0f6444"}
Oct 31 03:45:01.074749 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:01.074660    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:45:01.900518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:01.900468    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerStarted","Data":"dc6c350374d47ed43ae421c9375287cf54b8ebb222f9f06e37570f21d7aa13ec"}
Oct 31 03:45:01.919278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:01.919208    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" podStartSLOduration=1.919194643 podStartE2EDuration="1.919194643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:01.918965522 +0000 UTC m=+1776.444185112" watchObservedRunningTime="2026-10-31 03:45:01.919194643 +0000 UTC m=+1776.444414233"
Oct 31 03:45:02.958402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:02.954171    5098 generic.go:356] "Generic (PLEG): container finished" podID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerID="dc6c350374d47ed43ae421c9375287cf54b8ebb222f9f06e37570f21d7aa13ec" exitCode=60
Oct 31 03:45:02.958402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:02.954479    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerDied","Data":"dc6c350374d47ed43ae421c9375287cf54b8ebb222f9f06e37570f21d7aa13ec"}
Oct 31 03:45:02.958402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:02.955062    5098 scope.go:122] "RemoveContainer" containerID="dc6c350374d47ed43ae421c9375287cf54b8ebb222f9f06e37570f21d7aa13ec"
Oct 31 03:45:03.966899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:03.966864    5098 generic.go:356] "Generic (PLEG): container finished" podID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerID="1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8" exitCode=60
Oct 31 03:45:03.967365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:03.966942    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerDied","Data":"1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8"}
Oct 31 03:45:03.967365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:03.967006    5098 scope.go:122] "RemoveContainer" containerID="dc6c350374d47ed43ae421c9375287cf54b8ebb222f9f06e37570f21d7aa13ec"
Oct 31 03:45:03.967460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:03.967398    5098 scope.go:122] "RemoveContainer" containerID="1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8"
Oct 31 03:45:03.967893 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:03.967866    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6_odh-ai-gateway-infra(1e061a0a-a744-4f1a-bade-0342abeadccc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc"
Oct 31 03:45:04.979106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:04.979077    5098 scope.go:122] "RemoveContainer" containerID="1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8"
Oct 31 03:45:04.979523 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:04.979467    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6_odh-ai-gateway-infra(1e061a0a-a744-4f1a-bade-0342abeadccc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc"
Oct 31 03:45:14.605509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:14.605472    5098 scope.go:122] "RemoveContainer" containerID="1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8"
Oct 31 03:45:16.086449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:16.086410    5098 generic.go:356] "Generic (PLEG): container finished" podID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerID="ce2eab7b3a1b5fe6df7d5deb340a887c35a098fcf41c923c3c3e5561d0726a5e" exitCode=60
Oct 31 03:45:16.086997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:16.086486    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerDied","Data":"ce2eab7b3a1b5fe6df7d5deb340a887c35a098fcf41c923c3c3e5561d0726a5e"}
Oct 31 03:45:16.086997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:16.086569    5098 scope.go:122] "RemoveContainer" containerID="1dd69111a909111809e9e72cecd8337bc19d7edc6cb74e17f23ae03959ae3bd8"
Oct 31 03:45:16.087149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:16.087132    5098 scope.go:122] "RemoveContainer" containerID="ce2eab7b3a1b5fe6df7d5deb340a887c35a098fcf41c923c3c3e5561d0726a5e"
Oct 31 03:45:16.088875 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:16.087620    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6_odh-ai-gateway-infra(1e061a0a-a744-4f1a-bade-0342abeadccc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc"
Oct 31 03:45:17.121662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.121625    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:17.335465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.335427    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:17.496985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.496895    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:17.498903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.498843    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.498903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.498901    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.499150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.498983    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.499150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.498992    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.499271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.499211    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.499271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.499225    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.499271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.499241    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:17.509107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.509075    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:17.509294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.509155    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.537864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.537826    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\") pod \"1e061a0a-a744-4f1a-bade-0342abeadccc\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") "
Oct 31 03:45:17.538111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.538091    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\" (UniqueName: \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\") pod \"1e061a0a-a744-4f1a-bade-0342abeadccc\" (UID: \"1e061a0a-a744-4f1a-bade-0342abeadccc\") "
Oct 31 03:45:17.538288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.538253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca" pod "1e061a0a-a744-4f1a-bade-0342abeadccc" (UID: "1e061a0a-a744-4f1a-bade-0342abeadccc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:17.539060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.539021    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e061a0a-a744-4f1a-bade-0342abeadccc-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:17.540585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.540556    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh" pod "1e061a0a-a744-4f1a-bade-0342abeadccc" (UID: "1e061a0a-a744-4f1a-bade-0342abeadccc"). InnerVolumeSpecName "kube-api-access-ls6wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:17.587179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.587134    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:45:17.678662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.674870    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvbs\" (UniqueName: \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.678662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.675052    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.678662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.675509    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls6wh\" (UniqueName: \"kubernetes.io/projected/1e061a0a-a744-4f1a-bade-0342abeadccc-kube-api-access-ls6wh\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:17.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.792371    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvbs\" (UniqueName: \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.792542    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.793529    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:17.973117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:17.972983    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvbs\" (UniqueName: \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:18.108280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.108179    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6" event={"ID":"1e061a0a-a744-4f1a-bade-0342abeadccc","Type":"ContainerDied","Data":"8b4f2ac67e358b7eeaa7e6127fe3af137f5f5ac15647d0bf369ba5bdfa0f6444"}
Oct 31 03:45:18.108280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.108204    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"
Oct 31 03:45:18.108280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.108254    5098 scope.go:122] "RemoveContainer" containerID="ce2eab7b3a1b5fe6df7d5deb340a887c35a098fcf41c923c3c3e5561d0726a5e"
Oct 31 03:45:18.128936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.128906    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:18.142975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.142937    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:18.157994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.157949    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-34cb76-29890305-rtww6"]
Oct 31 03:45:18.678519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:18.678481    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:19.120053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:19.120013    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerStarted","Data":"f14cfc736cbd77f14bc637bf5a52d110d455e2bb0136a1eff72e55cb5872c395"}
Oct 31 03:45:20.207750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:20.204761    5098 generic.go:356] "Generic (PLEG): container finished" podID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerID="b2ee45f1b98bcd5a1c2f4e616367734cbab8aba9ee29b1145a0633852245d1d6" exitCode=60
Oct 31 03:45:20.207750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:20.205439    5098 scope.go:122] "RemoveContainer" containerID="b2ee45f1b98bcd5a1c2f4e616367734cbab8aba9ee29b1145a0633852245d1d6"
Oct 31 03:45:20.364290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:20.364246    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" path="/var/lib/kubelet/pods/1e061a0a-a744-4f1a-bade-0342abeadccc/volumes"
Oct 31 03:45:20.364862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:20.364827    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerDied","Data":"b2ee45f1b98bcd5a1c2f4e616367734cbab8aba9ee29b1145a0633852245d1d6"}
Oct 31 03:45:21.241669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:21.234516    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerStarted","Data":"d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"}
Oct 31 03:45:21.291730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:21.291564    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" podStartSLOduration=21.291545019 podStartE2EDuration="21.291545019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:21.279035584 +0000 UTC m=+1795.804255223" watchObservedRunningTime="2026-10-31 03:45:21.291545019 +0000 UTC m=+1795.816764610"
Oct 31 03:45:22.245382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:22.245342    5098 generic.go:356] "Generic (PLEG): container finished" podID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerID="d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948" exitCode=60
Oct 31 03:45:22.245862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:22.245436    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerDied","Data":"d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"}
Oct 31 03:45:22.245862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:22.245506    5098 scope.go:122] "RemoveContainer" containerID="b2ee45f1b98bcd5a1c2f4e616367734cbab8aba9ee29b1145a0633852245d1d6"
Oct 31 03:45:22.245862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:22.245782    5098 scope.go:122] "RemoveContainer" containerID="d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"
Oct 31 03:45:22.246235 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:22.246204    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt_odh-ai-gateway-infra(f06ad9e3-92cf-4ead-aa2e-f8fa630306b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4"
Oct 31 03:45:23.256593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:23.256565    5098 scope.go:122] "RemoveContainer" containerID="d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"
Oct 31 03:45:23.257032 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:23.256995    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt_odh-ai-gateway-infra(f06ad9e3-92cf-4ead-aa2e-f8fa630306b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4"
Oct 31 03:45:31.059758 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:31.059709    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:45:32.691333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:32.691306    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:45:32.763102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:32.763074    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:45:35.065934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:35.065902    5098 scope.go:122] "RemoveContainer" containerID="9890bed15cece313089768d65d02c8a919b464db55baafbab63c1c6372e9cb8c"
Oct 31 03:45:35.610514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:35.610482    5098 scope.go:122] "RemoveContainer" containerID="d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"
Oct 31 03:45:36.498751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:36.498660    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerStarted","Data":"2663a1f8b4e406c46c03d38022f5705a4bc3117de95567c5011c22a887b7bf20"}
Oct 31 03:45:37.510358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:37.510321    5098 generic.go:356] "Generic (PLEG): container finished" podID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerID="2663a1f8b4e406c46c03d38022f5705a4bc3117de95567c5011c22a887b7bf20" exitCode=60
Oct 31 03:45:37.510895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:37.510362    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerDied","Data":"2663a1f8b4e406c46c03d38022f5705a4bc3117de95567c5011c22a887b7bf20"}
Oct 31 03:45:37.510895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:37.510439    5098 scope.go:122] "RemoveContainer" containerID="d566bb99ab0c7a4459be08bfe4de7488dad6c05024b3197b0d6cef92a123e948"
Oct 31 03:45:37.512549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:37.511710    5098 scope.go:122] "RemoveContainer" containerID="2663a1f8b4e406c46c03d38022f5705a4bc3117de95567c5011c22a887b7bf20"
Oct 31 03:45:37.512549 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:37.512158    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt_odh-ai-gateway-infra(f06ad9e3-92cf-4ead-aa2e-f8fa630306b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4"
Oct 31 03:45:37.532481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:37.532447    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:38.711295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.711267    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:38.741247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.741213    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:38.742166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742141    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742162    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742186    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742192    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742205    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742210    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742222    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:38.742376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742226    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e061a0a-a744-4f1a-bade-0342abeadccc" containerName="cleanup"
Oct 31 03:45:38.742685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742383    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742396    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.742685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.742406    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" containerName="cleanup"
Oct 31 03:45:38.746502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.746477    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.765719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.765688    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:38.812152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.812112    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\") pod \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") "
Oct 31 03:45:38.812393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.812213    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\" (UniqueName: \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\") pod \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\" (UID: \"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4\") "
Oct 31 03:45:38.812673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.812647    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.812798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.812685    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdzx\" (UniqueName: \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.813353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.813324    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca" pod "f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" (UID: "f06ad9e3-92cf-4ead-aa2e-f8fa630306b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:38.816427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.816391    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs" pod "f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" (UID: "f06ad9e3-92cf-4ead-aa2e-f8fa630306b4"). InnerVolumeSpecName "kube-api-access-rhvbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:38.918631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.914831    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.918631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.914879    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdzx\" (UniqueName: \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.918631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.915406    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:38.918631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.915431    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvbs\" (UniqueName: \"kubernetes.io/projected/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4-kube-api-access-rhvbs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:38.918631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.916543    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:38.976165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:38.975679    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdzx\" (UniqueName: \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\") pod \"maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:39.078679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.078562    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:39.365796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.365758    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:39.365948 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:45:39.365863    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82927ab8_2b0d_4840_89fc_2670a23bf94a.slice/crio-c799de7942a69277e114ac0428091c7baf20d6798bebd462f40a6b879900605d WatchSource:0}: Error finding container c799de7942a69277e114ac0428091c7baf20d6798bebd462f40a6b879900605d: Status 404 returned error can't find the container with id c799de7942a69277e114ac0428091c7baf20d6798bebd462f40a6b879900605d
Oct 31 03:45:39.533869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.533837    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"
Oct 31 03:45:39.534064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.533875    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt" event={"ID":"f06ad9e3-92cf-4ead-aa2e-f8fa630306b4","Type":"ContainerDied","Data":"f14cfc736cbd77f14bc637bf5a52d110d455e2bb0136a1eff72e55cb5872c395"}
Oct 31 03:45:39.534064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.533953    5098 scope.go:122] "RemoveContainer" containerID="2663a1f8b4e406c46c03d38022f5705a4bc3117de95567c5011c22a887b7bf20"
Oct 31 03:45:39.535651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.535617    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerStarted","Data":"c799de7942a69277e114ac0428091c7baf20d6798bebd462f40a6b879900605d"}
Oct 31 03:45:39.565234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.565200    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:39.575015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.574978    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5396c2-29890305-ds2xt"]
Oct 31 03:45:39.614420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:39.614334    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06ad9e3-92cf-4ead-aa2e-f8fa630306b4" path="/var/lib/kubelet/pods/f06ad9e3-92cf-4ead-aa2e-f8fa630306b4/volumes"
Oct 31 03:45:40.638507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:40.637362    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerStarted","Data":"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"}
Oct 31 03:45:40.638507 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:40.637963    5098 scope.go:122] "RemoveContainer" containerID="7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"
Oct 31 03:45:41.649470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649432    5098 generic.go:356] "Generic (PLEG): container finished" podID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerID="7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030" exitCode=60
Oct 31 03:45:41.649470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649463    5098 generic.go:356] "Generic (PLEG): container finished" podID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerID="5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e" exitCode=60
Oct 31 03:45:41.650040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649521    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerDied","Data":"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"}
Oct 31 03:45:41.650040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649570    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerDied","Data":"5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e"}
Oct 31 03:45:41.650040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649598    5098 scope.go:122] "RemoveContainer" containerID="7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"
Oct 31 03:45:41.650040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.649931    5098 scope.go:122] "RemoveContainer" containerID="5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e"
Oct 31 03:45:41.650278 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:41.650256    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd_odh-ai-gateway-infra(82927ab8-2b0d-4840-89fc-2670a23bf94a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a"
Oct 31 03:45:41.668312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.668281    5098 scope.go:122] "RemoveContainer" containerID="7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"
Oct 31 03:45:41.669095 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:41.669050    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030\": container with ID starting with 7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030 not found: ID does not exist" containerID="7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"
Oct 31 03:45:41.669315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:41.669107    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030"} err="failed to get container status \"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030\": rpc error: code = NotFound desc = could not find container \"7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030\": container with ID starting with 7c9eb2c846bcd829c06877e651cb986bdea88a5eb0b04055aaf7497a278de030 not found: ID does not exist"
Oct 31 03:45:42.689151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:42.689117    5098 scope.go:122] "RemoveContainer" containerID="5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e"
Oct 31 03:45:42.689671 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:42.689586    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd_odh-ai-gateway-infra(82927ab8-2b0d-4840-89fc-2670a23bf94a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a"
Oct 31 03:45:52.604637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:52.603916    5098 scope.go:122] "RemoveContainer" containerID="5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e"
Oct 31 03:45:53.815938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:53.815843    5098 generic.go:356] "Generic (PLEG): container finished" podID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerID="7dfd4b88583c41a99126828610793c14964de54bfa3a26570a0e0ab70d2c6b5f" exitCode=60
Oct 31 03:45:53.815938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:53.815908    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerDied","Data":"7dfd4b88583c41a99126828610793c14964de54bfa3a26570a0e0ab70d2c6b5f"}
Oct 31 03:45:53.816480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:53.815978    5098 scope.go:122] "RemoveContainer" containerID="5f0ced245e2c192a0d7262deec04d5a2ce20b43c92bdf97eee2c41cf6363bd8e"
Oct 31 03:45:53.816480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:53.816445    5098 scope.go:122] "RemoveContainer" containerID="7dfd4b88583c41a99126828610793c14964de54bfa3a26570a0e0ab70d2c6b5f"
Oct 31 03:45:53.816929 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:53.816901    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd_odh-ai-gateway-infra(82927ab8-2b0d-4840-89fc-2670a23bf94a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a"
Oct 31 03:45:54.848294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:54.848259    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:55.035204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.035177    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:55.067901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.067865    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:45:55.068765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068744    5098 cpu_manager.go:414] "containerMap: removing container" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.068765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068761    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.068971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068781    5098 cpu_manager.go:414] "containerMap: removing container" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.068971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068786    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.068971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068807    5098 cpu_manager.go:414] "containerMap: removing container" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.068971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068813    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.069187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.068988    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.069187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.069000    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.069187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.069011    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" containerName="cleanup"
Oct 31 03:45:55.074243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.073160    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.081618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.081450    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:45:55.156990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.156899    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\" (UniqueName: \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\") pod \"82927ab8-2b0d-4840-89fc-2670a23bf94a\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") "
Oct 31 03:45:55.157206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.157036    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\") pod \"82927ab8-2b0d-4840-89fc-2670a23bf94a\" (UID: \"82927ab8-2b0d-4840-89fc-2670a23bf94a\") "
Oct 31 03:45:55.157408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.157381    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca" pod "82927ab8-2b0d-4840-89fc-2670a23bf94a" (UID: "82927ab8-2b0d-4840-89fc-2670a23bf94a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:45:55.157711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.157689    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv5jj\" (UniqueName: \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.157774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.157765    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.157922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.157911    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/82927ab8-2b0d-4840-89fc-2670a23bf94a-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:55.159172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.159154    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx" pod "82927ab8-2b0d-4840-89fc-2670a23bf94a" (UID: "82927ab8-2b0d-4840-89fc-2670a23bf94a"). InnerVolumeSpecName "kube-api-access-qpdzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:45:55.258998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.258957    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv5jj\" (UniqueName: \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.259188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.259039    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.259248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.259229    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdzx\" (UniqueName: \"kubernetes.io/projected/82927ab8-2b0d-4840-89fc-2670a23bf94a-kube-api-access-qpdzx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:45:55.259771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.259749    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.268918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.268894    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv5jj\" (UniqueName: \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.400267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.400234    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:45:55.697893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.696858    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:45:55.837438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.837316    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" event={"ID":"91d797f0-4d67-46c3-841d-8003745c163f","Type":"ContainerStarted","Data":"0c72d0aea3e0cc0a00a6c9d764d5614a427e001c70377cc3ba1770e87fae7d1e"}
Oct 31 03:45:55.839190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.839162    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd" event={"ID":"82927ab8-2b0d-4840-89fc-2670a23bf94a","Type":"ContainerDied","Data":"c799de7942a69277e114ac0428091c7baf20d6798bebd462f40a6b879900605d"}
Oct 31 03:45:55.839190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.839181    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"
Oct 31 03:45:55.839430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.839215    5098 scope.go:122] "RemoveContainer" containerID="7dfd4b88583c41a99126828610793c14964de54bfa3a26570a0e0ab70d2c6b5f"
Oct 31 03:45:55.867918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.867875    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:55.876479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:55.876443    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d86a77-29890305-9mdsd"]
Oct 31 03:45:56.849876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:56.849784    5098 generic.go:356] "Generic (PLEG): container finished" podID="91d797f0-4d67-46c3-841d-8003745c163f" containerID="593c616dff55e895eda79eec648e16c65ba5328273a588298578195e6ffe19c0" exitCode=60
Oct 31 03:45:56.850078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:56.849877    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" event={"ID":"91d797f0-4d67-46c3-841d-8003745c163f","Type":"ContainerDied","Data":"593c616dff55e895eda79eec648e16c65ba5328273a588298578195e6ffe19c0"}
Oct 31 03:45:56.850262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:56.850242    5098 scope.go:122] "RemoveContainer" containerID="593c616dff55e895eda79eec648e16c65ba5328273a588298578195e6ffe19c0"
Oct 31 03:45:57.613852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:57.613818    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82927ab8-2b0d-4840-89fc-2670a23bf94a" path="/var/lib/kubelet/pods/82927ab8-2b0d-4840-89fc-2670a23bf94a/volumes"
Oct 31 03:45:57.934628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:57.926239    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" event={"ID":"91d797f0-4d67-46c3-841d-8003745c163f","Type":"ContainerStarted","Data":"34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69"}
Oct 31 03:45:58.006142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:58.006073    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" podStartSLOduration=58.006053068 podStartE2EDuration="58.006053068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:45:57.965405834 +0000 UTC m=+1832.490625525" watchObservedRunningTime="2026-10-31 03:45:58.006053068 +0000 UTC m=+1832.531272658"
Oct 31 03:45:58.974408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:58.974367    5098 generic.go:356] "Generic (PLEG): container finished" podID="91d797f0-4d67-46c3-841d-8003745c163f" containerID="34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69" exitCode=60
Oct 31 03:45:58.980737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:58.975053    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" event={"ID":"91d797f0-4d67-46c3-841d-8003745c163f","Type":"ContainerDied","Data":"34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69"}
Oct 31 03:45:58.980737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:58.980120    5098 scope.go:122] "RemoveContainer" containerID="593c616dff55e895eda79eec648e16c65ba5328273a588298578195e6ffe19c0"
Oct 31 03:45:58.980737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:58.980253    5098 scope.go:122] "RemoveContainer" containerID="34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69"
Oct 31 03:45:58.981096 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:58.980775    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk_odh-ai-gateway-infra(91d797f0-4d67-46c3-841d-8003745c163f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" podUID="91d797f0-4d67-46c3-841d-8003745c163f"
Oct 31 03:45:59.985433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:45:59.985404    5098 scope.go:122] "RemoveContainer" containerID="34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69"
Oct 31 03:45:59.985890 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:45:59.985738    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk_odh-ai-gateway-infra(91d797f0-4d67-46c3-841d-8003745c163f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" podUID="91d797f0-4d67-46c3-841d-8003745c163f"
Oct 31 03:46:00.146718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:00.146551    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:46:01.065743 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:46:01.065711    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:46:01.297625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.297578    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:46:01.321101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.321068    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\" (UniqueName: \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\") pod \"91d797f0-4d67-46c3-841d-8003745c163f\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") "
Oct 31 03:46:01.321360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.321248    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\") pod \"91d797f0-4d67-46c3-841d-8003745c163f\" (UID: \"91d797f0-4d67-46c3-841d-8003745c163f\") "
Oct 31 03:46:01.321846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.321819    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca" pod "91d797f0-4d67-46c3-841d-8003745c163f" (UID: "91d797f0-4d67-46c3-841d-8003745c163f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:01.322161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.322140    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d797f0-4d67-46c3-841d-8003745c163f-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:01.326451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.326415    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj" pod "91d797f0-4d67-46c3-841d-8003745c163f" (UID: "91d797f0-4d67-46c3-841d-8003745c163f"). InnerVolumeSpecName "kube-api-access-rv5jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:01.423699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:01.423654    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rv5jj\" (UniqueName: \"kubernetes.io/projected/91d797f0-4d67-46c3-841d-8003745c163f-kube-api-access-rv5jj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:02.007362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:02.007260    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk" event={"ID":"91d797f0-4d67-46c3-841d-8003745c163f","Type":"ContainerDied","Data":"0c72d0aea3e0cc0a00a6c9d764d5614a427e001c70377cc3ba1770e87fae7d1e"}
Oct 31 03:46:02.007362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:02.007343    5098 scope.go:122] "RemoveContainer" containerID="34a068d0f86d3ab0d0eee2666efc1a55ec2e11cd9f0bfde3716c78d585d1bf69"
Oct 31 03:46:02.007596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:02.007277    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"
Oct 31 03:46:02.031657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:02.031621    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:46:02.039397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:02.039349    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-86a834-29890305-4l6lk"]
Oct 31 03:46:03.615778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:03.615742    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91d797f0-4d67-46c3-841d-8003745c163f" path="/var/lib/kubelet/pods/91d797f0-4d67-46c3-841d-8003745c163f/volumes"
Oct 31 03:46:09.578778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.578682    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"]
Oct 31 03:46:09.579659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579591    5098 cpu_manager.go:414] "containerMap: removing container" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.579659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579625    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.579659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579639    5098 cpu_manager.go:414] "containerMap: removing container" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.579659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579645    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.579842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579806    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.579842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.579826    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="91d797f0-4d67-46c3-841d-8003745c163f" containerName="cleanup"
Oct 31 03:46:09.585648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.585579    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.592419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.591788    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:46:09.592419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.592096    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"]
Oct 31 03:46:09.593144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.593118    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:46:09.688255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.685945    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.688255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.686027    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwpmg\" (UniqueName: \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.688255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.686223    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.787802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.787758    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.788090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.788071    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.788141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.788131    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwpmg\" (UniqueName: \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.788569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.788541    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.790653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.790628    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.796510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.796481    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwpmg\" (UniqueName: \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\") pod \"maas-api-revoke-sub-8979a606-lsv7l\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:09.908211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:09.908120    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:10.199887 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:46:10.199854    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode558e83f_259e_4c74_997c_efe23a24039b.slice/crio-a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179 WatchSource:0}: Error finding container a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179: Status 404 returned error can't find the container with id a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179
Oct 31 03:46:10.200312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:10.200288    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"]
Oct 31 03:46:11.182573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:11.182531    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l" event={"ID":"e558e83f-259e-4c74-997c-efe23a24039b","Type":"ContainerStarted","Data":"277daf68db40c59a6b980a83dadbcc7ac23b139561e4d23c2b76592fa10eda02"}
Oct 31 03:46:11.183233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:11.182618    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l" event={"ID":"e558e83f-259e-4c74-997c-efe23a24039b","Type":"ContainerStarted","Data":"a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179"}
Oct 31 03:46:11.204219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:11.204159    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l" podStartSLOduration=2.204141646 podStartE2EDuration="2.204141646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:11.19933423 +0000 UTC m=+1845.724553820" watchObservedRunningTime="2026-10-31 03:46:11.204141646 +0000 UTC m=+1845.729361260"
Oct 31 03:46:11.256526 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:46:11.256491    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode558e83f_259e_4c74_997c_efe23a24039b.slice/crio-277daf68db40c59a6b980a83dadbcc7ac23b139561e4d23c2b76592fa10eda02.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:46:12.192966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:12.192929    5098 generic.go:356] "Generic (PLEG): container finished" podID="e558e83f-259e-4c74-997c-efe23a24039b" containerID="277daf68db40c59a6b980a83dadbcc7ac23b139561e4d23c2b76592fa10eda02" exitCode=0
Oct 31 03:46:12.193493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:12.193019    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l" event={"ID":"e558e83f-259e-4c74-997c-efe23a24039b","Type":"ContainerDied","Data":"277daf68db40c59a6b980a83dadbcc7ac23b139561e4d23c2b76592fa10eda02"}
Oct 31 03:46:13.500747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.500671    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:13.570417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.570372    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\" (UniqueName: \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\") pod \"e558e83f-259e-4c74-997c-efe23a24039b\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") "
Oct 31 03:46:13.570685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.570584    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\") pod \"e558e83f-259e-4c74-997c-efe23a24039b\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") "
Oct 31 03:46:13.570819 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.570708    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\") pod \"e558e83f-259e-4c74-997c-efe23a24039b\" (UID: \"e558e83f-259e-4c74-997c-efe23a24039b\") "
Oct 31 03:46:13.571754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.571331    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca" pod "e558e83f-259e-4c74-997c-efe23a24039b" (UID: "e558e83f-259e-4c74-997c-efe23a24039b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:13.571754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.571710    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:13.575663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.574362    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg" pod "e558e83f-259e-4c74-997c-efe23a24039b" (UID: "e558e83f-259e-4c74-997c-efe23a24039b"). InnerVolumeSpecName "kube-api-access-gwpmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:13.575663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.574514    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca" pod "e558e83f-259e-4c74-997c-efe23a24039b" (UID: "e558e83f-259e-4c74-997c-efe23a24039b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:13.673265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.673224    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e558e83f-259e-4c74-997c-efe23a24039b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:13.673265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:13.673268    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwpmg\" (UniqueName: \"kubernetes.io/projected/e558e83f-259e-4c74-997c-efe23a24039b-kube-api-access-gwpmg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:14.257900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:14.257867    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"
Oct 31 03:46:14.258111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:14.257897    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l" event={"ID":"e558e83f-259e-4c74-997c-efe23a24039b","Type":"ContainerDied","Data":"a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179"}
Oct 31 03:46:14.258111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:14.257945    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9f6418acb0497b03aa29a3a41dd1d94065f53e5614d78dcae3a0f74c9e3f179"
Oct 31 03:46:21.131203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.131164    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"]
Oct 31 03:46:21.132088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.132066    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e558e83f-259e-4c74-997c-efe23a24039b" containerName="revoke-keys"
Oct 31 03:46:21.132088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.132086    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e558e83f-259e-4c74-997c-efe23a24039b" containerName="revoke-keys"
Oct 31 03:46:21.132253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.132243    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e558e83f-259e-4c74-997c-efe23a24039b" containerName="revoke-keys"
Oct 31 03:46:21.136559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.136530    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.140324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.140297    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:46:21.141668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.141645    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:46:21.144084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.144058    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"]
Oct 31 03:46:21.172183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.172147    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.172398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.172309    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbwtw\" (UniqueName: \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.172480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.172412    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.273627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.273568    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbwtw\" (UniqueName: \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.273836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.273707    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.273953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.273936    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.274580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.274553    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.276251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.276230    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.282445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.282420    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbwtw\" (UniqueName: \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\") pod \"maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.456973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.456891    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:21.748061 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:46:21.748028    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f583af3_f2ab_4e96_a1eb_abc0908beda3.slice/crio-76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679 WatchSource:0}: Error finding container 76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679: Status 404 returned error can't find the container with id 76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679
Oct 31 03:46:21.748449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:21.748425    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"]
Oct 31 03:46:22.400198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:22.400151    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8" event={"ID":"1f583af3-f2ab-4e96-a1eb-abc0908beda3","Type":"ContainerStarted","Data":"0ad61034c8413af42ee8e9fa41515388af9ea6da6e49ad1f14709463146c658c"}
Oct 31 03:46:22.400646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:22.400262    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8" event={"ID":"1f583af3-f2ab-4e96-a1eb-abc0908beda3","Type":"ContainerStarted","Data":"76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679"}
Oct 31 03:46:22.418233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:22.418131    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8" podStartSLOduration=1.418116795 podStartE2EDuration="1.418116795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:22.417445434 +0000 UTC m=+1856.942665023" watchObservedRunningTime="2026-10-31 03:46:22.418116795 +0000 UTC m=+1856.943336384"
Oct 31 03:46:23.519898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:23.519858    5098 generic.go:356] "Generic (PLEG): container finished" podID="1f583af3-f2ab-4e96-a1eb-abc0908beda3" containerID="0ad61034c8413af42ee8e9fa41515388af9ea6da6e49ad1f14709463146c658c" exitCode=0
Oct 31 03:46:23.521309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:23.520572    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8" event={"ID":"1f583af3-f2ab-4e96-a1eb-abc0908beda3","Type":"ContainerDied","Data":"0ad61034c8413af42ee8e9fa41515388af9ea6da6e49ad1f14709463146c658c"}
Oct 31 03:46:24.805230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.805147    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:24.874768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.874733    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\") pod \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") "
Oct 31 03:46:24.874957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.874809    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\" (UniqueName: \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\") pod \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") "
Oct 31 03:46:24.875078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.875056    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\") pod \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\" (UID: \"1f583af3-f2ab-4e96-a1eb-abc0908beda3\") "
Oct 31 03:46:24.875144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.875115    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca" pod "1f583af3-f2ab-4e96-a1eb-abc0908beda3" (UID: "1f583af3-f2ab-4e96-a1eb-abc0908beda3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:24.875841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.875817    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:24.877658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.877586    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca" pod "1f583af3-f2ab-4e96-a1eb-abc0908beda3" (UID: "1f583af3-f2ab-4e96-a1eb-abc0908beda3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:24.877778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.877745    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw" pod "1f583af3-f2ab-4e96-a1eb-abc0908beda3" (UID: "1f583af3-f2ab-4e96-a1eb-abc0908beda3"). InnerVolumeSpecName "kube-api-access-nbwtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:24.977967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.977924    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f583af3-f2ab-4e96-a1eb-abc0908beda3-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:24.977967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:24.977958    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbwtw\" (UniqueName: \"kubernetes.io/projected/1f583af3-f2ab-4e96-a1eb-abc0908beda3-kube-api-access-nbwtw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:25.539554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:25.539516    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"
Oct 31 03:46:25.539554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:25.539548    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8" event={"ID":"1f583af3-f2ab-4e96-a1eb-abc0908beda3","Type":"ContainerDied","Data":"76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679"}
Oct 31 03:46:25.539805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:25.539596    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76fce7735da3bd26e4bdc0a5da534f778cf561829ca2adec8387a2ef3c704679"
Oct 31 03:46:31.061013 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:46:31.060970    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:46:33.776421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:33.776382    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:46:35.135640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:35.132078    5098 scope.go:122] "RemoveContainer" containerID="5e202ad9031899ace7a53e8089c64fe9d64cbd4820896b48e6defe1de352e26b"
Oct 31 03:46:35.207521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:35.207471    5098 scope.go:122] "RemoveContainer" containerID="3cd7ff93f3adefa844f5d04aa0e80200cebddd5847a971d59697a725912d9161"
Oct 31 03:46:35.274649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:35.273950    5098 scope.go:122] "RemoveContainer" containerID="6a56a0f220f3a9a2a3759a6432b0e76efeb3820aaafcb641c9bc77920f3f96d0"
Oct 31 03:46:36.465897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:36.465741    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:46:38.743232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:38.743177    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"]
Oct 31 03:46:38.749951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:38.749914    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-8385c5-qj4q8"]
Oct 31 03:46:39.619395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:39.619355    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f583af3-f2ab-4e96-a1eb-abc0908beda3" path="/var/lib/kubelet/pods/1f583af3-f2ab-4e96-a1eb-abc0908beda3/volumes"
Oct 31 03:46:46.492512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.492477    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"]
Oct 31 03:46:46.494137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.494110    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1f583af3-f2ab-4e96-a1eb-abc0908beda3" containerName="revoke-keys"
Oct 31 03:46:46.494137 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.494135    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f583af3-f2ab-4e96-a1eb-abc0908beda3" containerName="revoke-keys"
Oct 31 03:46:46.494380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.494365    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f583af3-f2ab-4e96-a1eb-abc0908beda3" containerName="revoke-keys"
Oct 31 03:46:46.505794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.505764    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.511086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.511048    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"]
Oct 31 03:46:46.526183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.526148    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:46:46.526559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.526410    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:46:46.600197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.600147    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.600385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.600349    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.600535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.600518    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vff72\" (UniqueName: \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.702513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.702474    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vff72\" (UniqueName: \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.702751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.702659    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.702751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.702744    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.703385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.703360    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.705250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.705232    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.711082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.711052    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vff72\" (UniqueName: \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\") pod \"maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:46.832155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:46.832122    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:47.151376 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:46:47.151342    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5e961e_1e9d_4680_a5fd_0597b74f260b.slice/crio-d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee WatchSource:0}: Error finding container d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee: Status 404 returned error can't find the container with id d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee
Oct 31 03:46:47.153762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:47.153732    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"]
Oct 31 03:46:48.147071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:48.147036    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m" event={"ID":"7f5e961e-1e9d-4680-a5fd-0597b74f260b","Type":"ContainerStarted","Data":"52452547b86d78337c2f8e52a9f8f3cce8061daa3c68e93efae92596bd57f43a"}
Oct 31 03:46:48.147489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:48.147091    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m" event={"ID":"7f5e961e-1e9d-4680-a5fd-0597b74f260b","Type":"ContainerStarted","Data":"d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee"}
Oct 31 03:46:48.170676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:48.170622    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m" podStartSLOduration=2.170592385 podStartE2EDuration="2.170592385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:46:48.164565376 +0000 UTC m=+1882.689784966" watchObservedRunningTime="2026-10-31 03:46:48.170592385 +0000 UTC m=+1882.695811973"
Oct 31 03:46:49.165900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:49.165866    5098 generic.go:356] "Generic (PLEG): container finished" podID="7f5e961e-1e9d-4680-a5fd-0597b74f260b" containerID="52452547b86d78337c2f8e52a9f8f3cce8061daa3c68e93efae92596bd57f43a" exitCode=0
Oct 31 03:46:49.166387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:49.165927    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m" event={"ID":"7f5e961e-1e9d-4680-a5fd-0597b74f260b","Type":"ContainerDied","Data":"52452547b86d78337c2f8e52a9f8f3cce8061daa3c68e93efae92596bd57f43a"}
Oct 31 03:46:50.353754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.353725    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:50.481772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.474650    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\") pod \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") "
Oct 31 03:46:50.481772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.474742    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\") pod \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") "
Oct 31 03:46:50.481772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.481534    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\" (UniqueName: \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\") pod \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\" (UID: \"7f5e961e-1e9d-4680-a5fd-0597b74f260b\") "
Oct 31 03:46:50.485349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.484990    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca" pod "7f5e961e-1e9d-4680-a5fd-0597b74f260b" (UID: "7f5e961e-1e9d-4680-a5fd-0597b74f260b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:46:50.489216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.489115    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72" pod "7f5e961e-1e9d-4680-a5fd-0597b74f260b" (UID: "7f5e961e-1e9d-4680-a5fd-0597b74f260b"). InnerVolumeSpecName "kube-api-access-vff72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:46:50.517127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.517072    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca" pod "7f5e961e-1e9d-4680-a5fd-0597b74f260b" (UID: "7f5e961e-1e9d-4680-a5fd-0597b74f260b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:46:50.586061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.586009    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.586061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.586058    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f5e961e-1e9d-4680-a5fd-0597b74f260b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:50.586061 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:50.586069    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vff72\" (UniqueName: \"kubernetes.io/projected/7f5e961e-1e9d-4680-a5fd-0597b74f260b-kube-api-access-vff72\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:46:51.185577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:51.185546    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"
Oct 31 03:46:51.185785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:51.185571    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m" event={"ID":"7f5e961e-1e9d-4680-a5fd-0597b74f260b","Type":"ContainerDied","Data":"d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee"}
Oct 31 03:46:51.185785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:46:51.185645    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d383b04e2d279fbd3e05fa2532074bdbf7afd637106e3b6233758e717df16fee"
Oct 31 03:47:01.059802 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:47:01.059762    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:47:02.944267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.944229    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"]
Oct 31 03:47:02.945656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.945625    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7f5e961e-1e9d-4680-a5fd-0597b74f260b" containerName="revoke-keys"
Oct 31 03:47:02.945656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.945653    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5e961e-1e9d-4680-a5fd-0597b74f260b" containerName="revoke-keys"
Oct 31 03:47:02.945951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.945934    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5e961e-1e9d-4680-a5fd-0597b74f260b" containerName="revoke-keys"
Oct 31 03:47:02.959788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.953813    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:02.960578 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.960544    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"]
Oct 31 03:47:02.966135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.966106    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:47:02.966321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.966181    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:02.983042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.983002    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:02.983256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.983193    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:02.983334 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:02.983320    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vk97\" (UniqueName: \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.085025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.084988    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.085248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.085135    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.085339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.085270    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vk97\" (UniqueName: \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.086033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.086004    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.088109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.088080    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.116302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.116265    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vk97\" (UniqueName: \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\") pod \"maas-api-revoke-sub-8cc9dfe5-n4jf9\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.291467 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.291420    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:03.596745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:03.596699    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"]
Oct 31 03:47:03.597172 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:03.597148    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2019f127_4844_447e_99af_c0e788bff310.slice/crio-9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5 WatchSource:0}: Error finding container 9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5: Status 404 returned error can't find the container with id 9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5
Oct 31 03:47:04.326521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:04.326476    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9" event={"ID":"2019f127-4844-447e-99af-c0e788bff310","Type":"ContainerStarted","Data":"306cd1ecfcc29e198063e33bbf4d46856f1cbbed73c20210f5e38ba1d08468bb"}
Oct 31 03:47:04.327092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:04.326545    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9" event={"ID":"2019f127-4844-447e-99af-c0e788bff310","Type":"ContainerStarted","Data":"9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5"}
Oct 31 03:47:04.349706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:04.349644    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9" podStartSLOduration=2.349623396 podStartE2EDuration="2.349623396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:04.345133829 +0000 UTC m=+1898.870353421" watchObservedRunningTime="2026-10-31 03:47:04.349623396 +0000 UTC m=+1898.874842984"
Oct 31 03:47:05.336873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:05.336831    5098 generic.go:356] "Generic (PLEG): container finished" podID="2019f127-4844-447e-99af-c0e788bff310" containerID="306cd1ecfcc29e198063e33bbf4d46856f1cbbed73c20210f5e38ba1d08468bb" exitCode=0
Oct 31 03:47:05.337341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:05.336906    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9" event={"ID":"2019f127-4844-447e-99af-c0e788bff310","Type":"ContainerDied","Data":"306cd1ecfcc29e198063e33bbf4d46856f1cbbed73c20210f5e38ba1d08468bb"}
Oct 31 03:47:06.548807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.548772    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:06.687442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.687359    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\") pod \"2019f127-4844-447e-99af-c0e788bff310\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") "
Oct 31 03:47:06.687442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.687405    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\") pod \"2019f127-4844-447e-99af-c0e788bff310\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") "
Oct 31 03:47:06.687715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.687556    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\" (UniqueName: \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\") pod \"2019f127-4844-447e-99af-c0e788bff310\" (UID: \"2019f127-4844-447e-99af-c0e788bff310\") "
Oct 31 03:47:06.687907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.687873    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca" pod "2019f127-4844-447e-99af-c0e788bff310" (UID: "2019f127-4844-447e-99af-c0e788bff310"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:06.688348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.688330    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2019f127-4844-447e-99af-c0e788bff310-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:06.689694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.689665    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca" pod "2019f127-4844-447e-99af-c0e788bff310" (UID: "2019f127-4844-447e-99af-c0e788bff310"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:06.689835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.689813    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97" pod "2019f127-4844-447e-99af-c0e788bff310" (UID: "2019f127-4844-447e-99af-c0e788bff310"). InnerVolumeSpecName "kube-api-access-2vk97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:06.790596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.790553    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2019f127-4844-447e-99af-c0e788bff310-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:06.790596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:06.790591    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vk97\" (UniqueName: \"kubernetes.io/projected/2019f127-4844-447e-99af-c0e788bff310-kube-api-access-2vk97\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:07.370352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:07.370316    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"
Oct 31 03:47:07.370535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:07.370351    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9" event={"ID":"2019f127-4844-447e-99af-c0e788bff310","Type":"ContainerDied","Data":"9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5"}
Oct 31 03:47:07.370535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:07.370439    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9095cd684aa5b20ee44abcabd706af6d4865c43e7c805e6e108875b8969354d5"
Oct 31 03:47:09.905700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:09.905658    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"]
Oct 31 03:47:09.912078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:09.912041    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6f28b4-ptc9m"]
Oct 31 03:47:11.614420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:11.614380    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5e961e-1e9d-4680-a5fd-0597b74f260b" path="/var/lib/kubelet/pods/7f5e961e-1e9d-4680-a5fd-0597b74f260b/volumes"
Oct 31 03:47:14.366721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.365251    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"]
Oct 31 03:47:14.366721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.366487    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2019f127-4844-447e-99af-c0e788bff310" containerName="revoke-keys"
Oct 31 03:47:14.366721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.366511    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2019f127-4844-447e-99af-c0e788bff310" containerName="revoke-keys"
Oct 31 03:47:14.367486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.366820    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2019f127-4844-447e-99af-c0e788bff310" containerName="revoke-keys"
Oct 31 03:47:14.392222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.392129    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"]
Oct 31 03:47:14.392402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.392280    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.403991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.398659    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:14.403991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.401822    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:47:14.502065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.502016    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.502261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.502217    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nbm\" (UniqueName: \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.502261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.502246    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.604157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.604114    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.604370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.604272    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nbm\" (UniqueName: \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.604370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.604298    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.605068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.605047    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.606574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.606551    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.619316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.619252    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nbm\" (UniqueName: \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\") pod \"maas-api-revoke-sub-f8a3b08b-4vspg\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:14.718863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:14.718827    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:15.018749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:15.018719    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"]
Oct 31 03:47:15.021838 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:15.021803    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19b11db7_1214_4449_8bb2_09d402b62d89.slice/crio-0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa WatchSource:0}: Error finding container 0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa: Status 404 returned error can't find the container with id 0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa
Oct 31 03:47:15.522396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:15.522360    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg" event={"ID":"19b11db7-1214-4449-8bb2-09d402b62d89","Type":"ContainerStarted","Data":"0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa"}
Oct 31 03:47:16.534473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:16.534437    5098 generic.go:356] "Generic (PLEG): container finished" podID="19b11db7-1214-4449-8bb2-09d402b62d89" containerID="a243195c02f8755b1347e8447dba82d098a15b636b3873a3f6ca99932560f326" exitCode=0
Oct 31 03:47:16.534953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:16.534517    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg" event={"ID":"19b11db7-1214-4449-8bb2-09d402b62d89","Type":"ContainerDied","Data":"a243195c02f8755b1347e8447dba82d098a15b636b3873a3f6ca99932560f326"}
Oct 31 03:47:17.946631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:17.944270    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:18.147678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.137907    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\" (UniqueName: \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\") pod \"19b11db7-1214-4449-8bb2-09d402b62d89\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") "
Oct 31 03:47:18.147678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.138078    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\") pod \"19b11db7-1214-4449-8bb2-09d402b62d89\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") "
Oct 31 03:47:18.147678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.138337    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\") pod \"19b11db7-1214-4449-8bb2-09d402b62d89\" (UID: \"19b11db7-1214-4449-8bb2-09d402b62d89\") "
Oct 31 03:47:18.147678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.140193    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca" pod "19b11db7-1214-4449-8bb2-09d402b62d89" (UID: "19b11db7-1214-4449-8bb2-09d402b62d89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:18.203893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.200861    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca" pod "19b11db7-1214-4449-8bb2-09d402b62d89" (UID: "19b11db7-1214-4449-8bb2-09d402b62d89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:18.204344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.202195    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm" pod "19b11db7-1214-4449-8bb2-09d402b62d89" (UID: "19b11db7-1214-4449-8bb2-09d402b62d89"). InnerVolumeSpecName "kube-api-access-w5nbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:18.248328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.248261    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:18.248328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.248293    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nbm\" (UniqueName: \"kubernetes.io/projected/19b11db7-1214-4449-8bb2-09d402b62d89-kube-api-access-w5nbm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:18.248328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.248304    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19b11db7-1214-4449-8bb2-09d402b62d89-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:18.568089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.568045    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg" event={"ID":"19b11db7-1214-4449-8bb2-09d402b62d89","Type":"ContainerDied","Data":"0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa"}
Oct 31 03:47:18.568318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.568122    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b483901e0ed4bd105563172e86b23d814799f3e9d6f7d9eb704d619ffa0cffa"
Oct 31 03:47:18.568318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:18.568066    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"
Oct 31 03:47:23.909347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:23.909312    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:47:23.909837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:23.909510    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerName="discovery" containerID="cri-o://2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22" gracePeriod=30
Oct 31 03:47:24.358103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.358075    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:47:24.401722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.401678    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:47:24.403041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403012    5098 cpu_manager.go:414] "containerMap: removing container" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerName="discovery"
Oct 31 03:47:24.403041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403039    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerName="discovery"
Oct 31 03:47:24.403272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403112    5098 cpu_manager.go:414] "containerMap: removing container" podUID="19b11db7-1214-4449-8bb2-09d402b62d89" containerName="revoke-keys"
Oct 31 03:47:24.403272 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403120    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19b11db7-1214-4449-8bb2-09d402b62d89" containerName="revoke-keys"
Oct 31 03:47:24.403376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403307    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerName="discovery"
Oct 31 03:47:24.403376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.403334    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="19b11db7-1214-4449-8bb2-09d402b62d89" containerName="revoke-keys"
Oct 31 03:47:24.433014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.432965    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433073    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433110    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433159    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433306    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433405    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.433514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.433464    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\") pod \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\" (UID: \"3ef938d9-71a9-4248-83b3-f2e5a204c28e\") "
Oct 31 03:47:24.435921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.435879    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:24.437283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.436938    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:24.442264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.442224    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:24.465875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.462077    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:47:24.467163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.467131    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.468352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.468313    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "kube-api-access-s5ght". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:24.471299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.471267    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:24.471552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.471528    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:24.476134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.476054    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-73e222-openshift-default-dockercfg-l6dr2\""
Oct 31 03:47:24.493045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.493001    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token" pod "3ef938d9-71a9-4248-83b3-f2e5a204c28e" (UID: "3ef938d9-71a9-4248-83b3-f2e5a204c28e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:24.496705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.496672    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:47:24.496945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.496917    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.523738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.522843    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:47:24.529668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.529582    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.531341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.531313    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:47:24.531701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.531671    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.535868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.535843    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536057    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536222    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfgz9\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536341    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536442    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536657    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536824    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.536934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.536924    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.537044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537031    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.537236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537226    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537305    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-dns-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537370    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-csr-ca-configmap\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537431    5098 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-cacerts\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537492    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5ght\" (UniqueName: \"kubernetes.io/projected/3ef938d9-71a9-4248-83b3-f2e5a204c28e-kube-api-access-s5ght\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537566    5098 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ef938d9-71a9-4248-83b3-f2e5a204c28e-istio-kubeconfig\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.537676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.537667    5098 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ef938d9-71a9-4248-83b3-f2e5a204c28e-local-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:24.539110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.539086    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:47:24.539207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.539132    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:47:24.539256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.539226    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.540259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.539931    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:47:24.546505 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.546481    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:47:24.551728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.551705    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:47:24.576206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.576174    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5d9b6-openshift-default-dockercfg-fjkqj\""
Oct 31 03:47:24.639366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639328    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.639366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639436    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639465    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639493    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639527    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639562    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639628    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.639725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639705    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rqsx\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639764    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639794    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639825    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639829    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.639853    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640059    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.640099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640093    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640117    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640163    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640210    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640305    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640376    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640430    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640463    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640490    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640547    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640589    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640622    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640749    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d7h5\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640844    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640889    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640914    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.640937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640927    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640971    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.640991    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gwj6\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641022    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641130    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641168    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641230    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.641380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641327    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641385    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641455    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641494    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641525    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcvn\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641590    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641698    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.641756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641740    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.642116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641782    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfgz9\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.642116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641811    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.642116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.641957    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.642116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.642019    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.643971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.643946    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.644239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.644217    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.646482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.646455    5098 generic.go:356] "Generic (PLEG): container finished" podID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" containerID="2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22" exitCode=0
Oct 31 03:47:24.646593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.646519    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"
Oct 31 03:47:24.646593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.646530    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" event={"ID":"3ef938d9-71a9-4248-83b3-f2e5a204c28e","Type":"ContainerDied","Data":"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"}
Oct 31 03:47:24.646593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.646585    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4" event={"ID":"3ef938d9-71a9-4248-83b3-f2e5a204c28e","Type":"ContainerDied","Data":"80055eddbad8eac786d42df2264227c5fe3c028f6f709048d70623ab5afa44d6"}
Oct 31 03:47:24.646739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.646647    5098 scope.go:122] "RemoveContainer" containerID="2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"
Oct 31 03:47:24.650924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.650901    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfgz9\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.651306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.651288    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\") pod \"e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") " pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.660510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.660473    5098 scope.go:122] "RemoveContainer" containerID="2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"
Oct 31 03:47:24.660888 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:47:24.660861    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22\": container with ID starting with 2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22 not found: ID does not exist" containerID="2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"
Oct 31 03:47:24.660983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.660900    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22"} err="failed to get container status \"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22\": rpc error: code = NotFound desc = could not find container \"2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22\": container with ID starting with 2c5c353e64827af60b6d944a57696f090adc5fd746a5bbfa93ce564ad5bc3c22 not found: ID does not exist"
Oct 31 03:47:24.677481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.677447    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:47:24.683392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.683360    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bczj4"]
Oct 31 03:47:24.743520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743481    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743531    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743554    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gwj6\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743575    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743618    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743642    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743678    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.743745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743734    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743770    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743822    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743854    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcvn\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743926    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.743965    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744005    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744031    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744064    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744120    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.744145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744146    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744160    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744188    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744215    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744252    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744292    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744347    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744395    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744440    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rqsx\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744474    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744494    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744530    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744619    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744653    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744671    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.744693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744699    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744724    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744752    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744886    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745127    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.744793    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745414    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745461    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745499    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.745574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745526    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.745585    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d7h5\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746115    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746182    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746265    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746295    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746315    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.746811    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747055    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747075    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747083    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747159    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747330    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747373    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747396    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.747858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.747575    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.749002 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.748979    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.749145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.748994    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.749253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.749229    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.749322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.749229    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.749322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.749284    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.749412 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.749343    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.749791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.749772    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.753147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.752841    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gwj6\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.753147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.753103    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcvn\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.753553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.753526    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.753846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.753829    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\") pod \"e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") " pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.754258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.754237    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.755056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.755034    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d7h5\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.755572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.755551    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.756032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.756010    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rqsx\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\") pod \"e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") " pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:24.860703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.853937    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:24.860703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.854760    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:24.867098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.867040    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:24.877098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.877057    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:24.908544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:24.908509    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:25.254499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.254463    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:47:25.264202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.263379    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.264202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.263485    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.264202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.263527    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.265215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.265018    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:47:25.268734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.267707    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.268734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.267798    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.268734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.267845    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.370265 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:25.370219    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14632dd8_04bf_453b_9f1b_2338355eeae6.slice/crio-c77dc71402b23de7f9e4e17fcb45ce02f2cd5d57eb81515e987a386055af73ac WatchSource:0}: Error finding container c77dc71402b23de7f9e4e17fcb45ce02f2cd5d57eb81515e987a386055af73ac: Status 404 returned error can't find the container with id c77dc71402b23de7f9e4e17fcb45ce02f2cd5d57eb81515e987a386055af73ac
Oct 31 03:47:25.376916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.374337    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:47:25.376916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.376271    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.376916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.376352    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.376916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.376393    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.525818 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:25.525759    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd9e3a9_0f94_493f_9d4c_18e6302ae53a.slice/crio-4890e398a0df71edc5cd1551749893c4a5a47ed6be0f9db93f7bafcd84837d98 WatchSource:0}: Error finding container 4890e398a0df71edc5cd1551749893c4a5a47ed6be0f9db93f7bafcd84837d98: Status 404 returned error can't find the container with id 4890e398a0df71edc5cd1551749893c4a5a47ed6be0f9db93f7bafcd84837d98
Oct 31 03:47:25.529451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.529409    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:47:25.529988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.529938    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.530081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.530032    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.530120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.530088    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.562971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.559970    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:47:25.565728 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:25.565679    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7936d4_6ad3_4216_990d_d7cc5d26902c.slice/crio-bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836 WatchSource:0}: Error finding container bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836: Status 404 returned error can't find the container with id bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836
Oct 31 03:47:25.573824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.573601    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.573824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.573730    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.573824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.573762    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:47:25.631562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.631521    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ef938d9-71a9-4248-83b3-f2e5a204c28e" path="/var/lib/kubelet/pods/3ef938d9-71a9-4248-83b3-f2e5a204c28e/volumes"
Oct 31 03:47:25.681396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.681351    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" event={"ID":"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a","Type":"ContainerStarted","Data":"4890e398a0df71edc5cd1551749893c4a5a47ed6be0f9db93f7bafcd84837d98"}
Oct 31 03:47:25.714052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.713348    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" event={"ID":"1f404570-36f6-458e-b0cd-b64a3ae53f89","Type":"ContainerStarted","Data":"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"}
Oct 31 03:47:25.714052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.713427    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" event={"ID":"1f404570-36f6-458e-b0cd-b64a3ae53f89","Type":"ContainerStarted","Data":"eeeeb8a600011fef5a9ab515092d331ce3cbd6aaad941c92ece71a08033d9ee8"}
Oct 31 03:47:25.719843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.717551    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" event={"ID":"95b89f6f-0c66-4c51-ae3a-821e1abe3b03","Type":"ContainerStarted","Data":"8ea0c035362c1f2b441ff38f48d52aafb390be049ad4f868fe4f34a9412cc58f"}
Oct 31 03:47:25.719843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.717633    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" event={"ID":"95b89f6f-0c66-4c51-ae3a-821e1abe3b03","Type":"ContainerStarted","Data":"b75ad4b2e0681f080f6f0fa33d8824cc2ecca00807156a04751bbf7ec80fd0c1"}
Oct 31 03:47:25.723466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.723426    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" event={"ID":"6d7936d4-6ad3-4216-990d-d7cc5d26902c","Type":"ContainerStarted","Data":"bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836"}
Oct 31 03:47:25.727046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.727015    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" event={"ID":"14632dd8-04bf-453b-9f1b-2338355eeae6","Type":"ContainerStarted","Data":"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"}
Oct 31 03:47:25.731069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.731028    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" event={"ID":"14632dd8-04bf-453b-9f1b-2338355eeae6","Type":"ContainerStarted","Data":"c77dc71402b23de7f9e4e17fcb45ce02f2cd5d57eb81515e987a386055af73ac"}
Oct 31 03:47:25.750822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.750758    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podStartSLOduration=239.750734181 podStartE2EDuration="3m59.750734181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:25.740699254 +0000 UTC m=+1920.265918843" watchObservedRunningTime="2026-10-31 03:47:25.750734181 +0000 UTC m=+1920.275953774"
Oct 31 03:47:25.791831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.791767    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podStartSLOduration=259.791739505 podStartE2EDuration="4m19.791739505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:25.778664829 +0000 UTC m=+1920.303884419" watchObservedRunningTime="2026-10-31 03:47:25.791739505 +0000 UTC m=+1920.316959095"
Oct 31 03:47:25.827627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.821273    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podStartSLOduration=254.82125432 podStartE2EDuration="4m14.82125432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:25.804797082 +0000 UTC m=+1920.330016673" watchObservedRunningTime="2026-10-31 03:47:25.82125432 +0000 UTC m=+1920.346473911"
Oct 31 03:47:25.849201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.849126    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podStartSLOduration=254.84910254 podStartE2EDuration="4m14.84910254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:25.838430984 +0000 UTC m=+1920.363650575" watchObservedRunningTime="2026-10-31 03:47:25.84910254 +0000 UTC m=+1920.374322206"
Oct 31 03:47:25.941592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.908745    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:25.944258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.944232    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:25.960733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.960684    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:25.960975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:25.960962    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:26.795324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.795264    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" event={"ID":"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a","Type":"ContainerStarted","Data":"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"}
Oct 31 03:47:26.803708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.800071    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" event={"ID":"6d7936d4-6ad3-4216-990d-d7cc5d26902c","Type":"ContainerStarted","Data":"1523f4d50f9ccf3bc50f38ec902426576925c93fd3221afd74cb6fdeeb8d2886"}
Oct 31 03:47:26.871172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.871131    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:26.881013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.880978    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:26.907986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.881294    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:26.907986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.886914    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:26.907986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.887333    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:26.907986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.887529    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:26.907986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.887564    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:26.912558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.909744    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:26.912558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.909811    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:26.913045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.912926    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:26.913045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.912984    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:27.018638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:26.923032    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podStartSLOduration=27.923006105 podStartE2EDuration="27.923006105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:46:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:26.826316237 +0000 UTC m=+1921.351535827" watchObservedRunningTime="2026-10-31 03:47:26.923006105 +0000 UTC m=+1921.448225698"
Oct 31 03:47:27.862085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.862040    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:27.877413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.877369    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:27.878064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.862716    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:27.878266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.878234    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:27.890329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.867627    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:27.890766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:27.890721    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:28.074974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.071719    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:28.074974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.071811    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:28.132722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.118448    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:28.132722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.118551    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:28.510381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.510102    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:47:28.595997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.595952    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"]
Oct 31 03:47:28.771115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.755097    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:28.831650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.805533    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:28.831650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.807006    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:47:28.832754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.832703    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:28.869985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.834652    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:28.870960 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.847439    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946000    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946345    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946386    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946445    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946465    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.946971    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.947003    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.956453    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.956720    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.956838    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7fq4\" (UniqueName: \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:28.993883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:28.978093    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"]
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.051191    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.051267    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.078428    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.082800    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.093979    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.094168    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.143635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.094259    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7fq4\" (UniqueName: \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.168215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.165717    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.223631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.222599    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:29.223631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.222677    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:29.317492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.303795    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.317492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.304461    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7fq4\" (UniqueName: \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\") pod \"maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.374629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.374043    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:29.863800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.861149    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:29.864176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.864129    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:29.864354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.864326    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:29.930629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.870423    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:30.026427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.020738    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:30.120692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.871569    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.871523    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.871473    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.121444    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:29.883727    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.121532    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.070999    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.121592    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:30.162634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.126280    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:47:30.357632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.352123    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:30.510921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.472719    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:30.510921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.510675    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566225    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566429    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566488    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566541    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566593    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:47:30.596074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.566674    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:47:30.896642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.872801    5098 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-nq5jp container/console-operator namespace/openshift-console-operator: Liveness probe status=failure output="Get \"https://10.132.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 03:47:30.896642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:30.872939    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console-operator/console-operator-7cd56b8dbc-nq5jp" podUID="1b4b412a-ef91-4c3e-89c3-9287445d404c" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.11:8443/healthz\": context deadline exceeded"
Oct 31 03:47:31.051635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.026945    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.081777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.081730    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:47:31.179085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.178965    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:47:31.534764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.185977    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" containerID="cri-o://2aafd46f3ceb4d244b016182aaaf742b49cda70ee0c5044a12b120f068682ce5" gracePeriod=30
Oct 31 03:47:31.556888 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.384196    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/health\": context deadline exceeded"
Oct 31 03:47:31.589229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.386144    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.657290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.475503    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.657586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.512496    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:31.657783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.512546    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.657932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.512600    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:47:31.693253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.512794    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-k66s8" podUID="17595e19-e8f1-47af-9b70-0b75e8559dde" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.700630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.513210    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.708747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.532988    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-8jbq7" podUID="9c5fe135-1668-41e1-a2d4-42aa4234e43e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.730637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.555837    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:31.757190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.757140    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:31.797574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.640219    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:31.918812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.693076    5098 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-6jnt8 container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded" start-of-body=
Oct 31 03:47:31.925465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:31.925408    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-6jnt8" podUID="5d7ff043-ea26-4c96-874d-c2d6ab18b6a9" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.132.0.29:8443/healthz\": context deadline exceeded"
Oct 31 03:47:32.027213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.027166    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:47:32.027518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.027480    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:32.046082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.028832    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:47:32.047802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.047678    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120149    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120217    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120332    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120356    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120419    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.121317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.120441    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:32.166638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.155226    5098 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-mv884 container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.132.0.28:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:47:32.166638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.155304    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-mv884" podUID="54e44444-9207-4253-aaa5-77b849d81581" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.28:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:32.166638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.156878    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.60:50052\" within 1s: context deadline exceeded"
Oct 31 03:47:32.166638 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:32.157027    5098 logging.go:55] [core] [Channel #4605 SubChannel #4606]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.60:50052", ServerName: "10.132.0.60:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.60:50052: operation was canceled"
Oct 31 03:47:32.349212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.348060    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.349212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.348125    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:32.353052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.349646    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.353052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.349699    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:32.353052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.350174    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.215:50052\" within 1s: context deadline exceeded"
Oct 31 03:47:32.458634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.423664    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.59:50052\" within 1s: context deadline exceeded"
Oct 31 03:47:32.486634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.462293    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.565727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.551134    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:32.565727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.502713    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:32.565727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.551233    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:32.641012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.608278    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/health\": context deadline exceeded"
Oct 31 03:47:32.750207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:32.750171    5098 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.173787028s: [/var/lib/containers/storage/overlay/fe69948cd3aa7e8437ff9331d24124e460a7a6170745d042667ebfb7105413fc/diff /var/log/pods/openshift-service-ca-operator_service-ca-operator-544d749c8c-vwf4f_61a64191-2d4c-48e4-bf08-e010b704cda5/service-ca-operator/1.log]; will not log again for this container unless duration exceeds 2s
Oct 31 03:47:33.288771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.288717    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:33.291413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.291375    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:47:33.317857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.306799    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:47:33.334192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.334152    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.334477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.334446    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:33.351575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.351527    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:33.353434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.353398    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:47:33.361756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.361696    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl" podUID="5899e70c-2b85-4866-aa96-3a755adfe0f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 31 03:47:33.379644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.376321    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.379644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.376383    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:33.379644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.377148    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.379644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.377185    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:33.532679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.464174    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.532679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.464239    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:33.561754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.533410    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.561754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.533484    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:33.596586 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:47:33.596474    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:47:33.695846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.695788    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:33.941382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.941282    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.941697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.941662    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:33.941928 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.941911    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:33.942043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:33.942025    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:34.021892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.021839    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.022220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.022169    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:34.051593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.051563    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.060967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.051879    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:34.075071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.075038    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.148645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.133637    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:34.304568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.304517    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:47:34.587687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.355880    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:34.587687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.356750    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:47:34.614130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.407705    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:47:34.640657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.640594    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:47:34.755856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.755796    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:34.800145 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:47:34.795139    5098 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="3.155s"
Oct 31 03:47:34.851705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.850992    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:34.881382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.881328    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.881682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.881648    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:34.891832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.891768    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:34.895186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.895152    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.895445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.895410    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:34.895671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.895651    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.895832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.895811    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:34.900911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.900838    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:34.901097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:34.901073    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:35.060558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.059811    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.060558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.059885    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:35.274643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.273899    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:47:35.274643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.273949    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98" event={"ID":"c402cc77-84ef-4713-9dc7-0f37a53ee135","Type":"ContainerStarted","Data":"7eb37302e49664bcb90fbb2e19fd937114eb9b1e12aa860c8f76209c5a4aad08"}
Oct 31 03:47:35.274643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.274016    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:47:35.274643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.274073    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl"
Oct 31 03:47:35.274643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.274098    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"]
Oct 31 03:47:35.632633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.599855    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.637543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.600062    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.697107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.600115    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.697844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.618277    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.698042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.658921    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.748648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.717158    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.748955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.717360    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.766058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.766012    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.867632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.852714    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:35.914632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.879825    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.949380    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.879871    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.949469    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.879902    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.949511    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.906356    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.949553    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.919958    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:35.998097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:35.949599    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:36.736247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.736194    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-l2r8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:47:36.736948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.736894    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:47:36.747044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.746995    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:36.780075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.772011    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:36.949631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.949244    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:47:36.963628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:36.953742    5098 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/health\": context deadline exceeded"
Oct 31 03:47:37.060807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.051758    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.060807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.051831    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:37.060807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.051927    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.060807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.051952    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:37.115631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.112087    5098 scope.go:122] "RemoveContainer" containerID="64b26c6e38d9b58d4a5350545165e1905a893c690556dc63f1409682ecadad5f"
Oct 31 03:47:37.207630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.116036    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.207630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.175324    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:37.207630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.194354    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": context deadline exceeded"
Oct 31 03:47:37.207630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.194838    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:47:37.207630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.195182    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": context deadline exceeded"
Oct 31 03:47:37.614556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.614512    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98" event={"ID":"c402cc77-84ef-4713-9dc7-0f37a53ee135","Type":"ContainerStarted","Data":"d92289d0546ec6fc79d6efb59c24a228b467976dc549cc89b661460853077207"}
Oct 31 03:47:37.768524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.768476    5098 scope.go:122] "RemoveContainer" containerID="40575fab36b47c3dfdaf87d49d096bda97ef7ead529eaf739241248718674251"
Oct 31 03:47:37.830658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.826901    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": context deadline exceeded"
Oct 31 03:47:37.909119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.909016    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.977503 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.977463    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": dial tcp 10.132.0.229:15021: connect: connection refused"
Oct 31 03:47:37.977728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.977403    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:37.977916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.920967    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 31 03:47:37.978200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.949815    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.978277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.978232    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused"
Oct 31 03:47:37.978337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.949885    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:37.978337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.978302    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:38.013902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.989456    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:38.013902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.989540    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:38.013902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.990150    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused" start-of-body=
Oct 31 03:47:38.013902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:37.990202    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.230:15021/healthz/ready\": dial tcp 10.132.0.230:15021: connect: connection refused"
Oct 31 03:47:38.040170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.019745    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:38.040170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.019846    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:38.040170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.020247    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body=
Oct 31 03:47:38.040170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.020301    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused"
Oct 31 03:47:38.041092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.041034    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.79:8000/ready\": context deadline exceeded"
Oct 31 03:47:38.196689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.191412    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98" podStartSLOduration=10.191390745 podStartE2EDuration="10.191390745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:38.14101026 +0000 UTC m=+1932.666229850" watchObservedRunningTime="2026-10-31 03:47:38.191390745 +0000 UTC m=+1932.716610338"
Oct 31 03:47:38.208880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.197049    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" probeResult="failure" output="Get \"https://10.132.0.108:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:38.208880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.197856    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:38.208880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.198156    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:38.636650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.633601    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 03:47:38.923432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.923302    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:47:38.937851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.937802    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:47:38.963082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.949530    5098 patch_prober.go:29] interesting pod/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body=
Oct 31 03:47:38.963367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.963338    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused"
Oct 31 03:47:38.971837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.971806    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 03:47:38.983686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.978059    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:38.983686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.979253    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:38.983686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.979409    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:47:38.983686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.979656    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln" podUID="511eac43-6d5d-4d2f-9c1e-ca4c515c9fea" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": context deadline exceeded"
Oct 31 03:47:38.983686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:38.980867    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:47:39.020936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.019566    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:47:39.050891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.050855    5098 generic.go:356] "Generic (PLEG): container finished" podID="c402cc77-84ef-4713-9dc7-0f37a53ee135" containerID="d92289d0546ec6fc79d6efb59c24a228b467976dc549cc89b661460853077207" exitCode=0
Oct 31 03:47:39.051268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.051244    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98" event={"ID":"c402cc77-84ef-4713-9dc7-0f37a53ee135","Type":"ContainerDied","Data":"d92289d0546ec6fc79d6efb59c24a228b467976dc549cc89b661460853077207"}
Oct 31 03:47:39.061620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.061560    5098 generic.go:356] "Generic (PLEG): container finished" podID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerID="2aafd46f3ceb4d244b016182aaaf742b49cda70ee0c5044a12b120f068682ce5" exitCode=0
Oct 31 03:47:39.062063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.061993    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" event={"ID":"2d5df961-8bd7-46bb-8a90-1129a3840c56","Type":"ContainerDied","Data":"2aafd46f3ceb4d244b016182aaaf742b49cda70ee0c5044a12b120f068682ce5"}
Oct 31 03:47:39.203687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.198846    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 31 03:47:39.210601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.210568    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:47:39.241405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.232626    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:39.241405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.237868    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:39.241405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.239904    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln"
Oct 31 03:47:39.291059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.280835    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:47:39.291059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.286255    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:39.291059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.286309    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:47:39.291059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.288394    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:39.297206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.297116    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:47:39.303030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.302996    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:39.304136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.303640    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:39.304752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.304733    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:47:39.354881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.354842    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.355076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.354903    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.355076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.354939    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358193    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358246    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358409    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358536    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358578    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.358907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.358694    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\") pod \"2d5df961-8bd7-46bb-8a90-1129a3840c56\" (UID: \"2d5df961-8bd7-46bb-8a90-1129a3840c56\") "
Oct 31 03:47:39.367318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.365362    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:39.367318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.365697    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:39.367318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.367284    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:39.371794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.367654    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:39.371794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.367997    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "kube-api-access-fp8sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:39.371794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.368203    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:39.371794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.370808    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:39.378757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.376017    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:39.378757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.376470    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:47:39.378757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.378050    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy"
Oct 31 03:47:39.378757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.378072    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy"
Oct 31 03:47:39.378757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.378331    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" containerName="istio-proxy"
Oct 31 03:47:39.392785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.392757    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.411279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.411237    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy" pod "2d5df961-8bd7-46bb-8a90-1129a3840c56" (UID: "2d5df961-8bd7-46bb-8a90-1129a3840c56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:39.411903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.411868    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-b8fd22\""
Oct 31 03:47:39.412994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.412972    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-b8fd22-dockercfg-brqfw\""
Oct 31 03:47:39.419771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.419733    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:47:39.430844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.430815    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:47:39.431079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.430965    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.439943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.439899    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:47:39.462669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.462634    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.462946 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.462907    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vswmz\" (UniqueName: \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.463414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.463350    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.463653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.463632    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.463963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.463945    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464056    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp8sh\" (UniqueName: \"kubernetes.io/projected/2d5df961-8bd7-46bb-8a90-1129a3840c56-kube-api-access-fp8sh\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464148    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464245 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464233    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5df961-8bd7-46bb-8a90-1129a3840c56-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464329    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464408    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464487    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464561    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.464686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.464672    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5df961-8bd7-46bb-8a90-1129a3840c56-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:39.646634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.569212    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.646634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.569277    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vswmz\" (UniqueName: \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.646634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.569511    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.646634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.569591    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.646634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.623425    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.693633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.668289    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vswmz\" (UniqueName: \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\") pod \"payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.733766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.705393    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.733766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.732532    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\") pod \"payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:39.734221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.733962    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:39.768032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:39.767990    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:40.073625 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.058514    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:40.092792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.092757    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:40.264403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.264362    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k" event={"ID":"2d5df961-8bd7-46bb-8a90-1129a3840c56","Type":"ContainerDied","Data":"65ddb4aaabc26e6582c60bd0b8a859afdd730c4288b31a8ee1b12773889d3337"}
Oct 31 03:47:40.266569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.266262    5098 scope.go:122] "RemoveContainer" containerID="2aafd46f3ceb4d244b016182aaaf742b49cda70ee0c5044a12b120f068682ce5"
Oct 31 03:47:40.266569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.264901    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"
Oct 31 03:47:40.307588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.307557    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:47:40.313397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.313360    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-l2r8k"]
Oct 31 03:47:40.518934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.518906    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:40.574993 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:40.574950    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078cad6a_6136_451f_9da0_9747a8b7dac4.slice/crio-2a46af755e068eb55e1d5c7db4b1eff65e0154d830358bc85949926739965551 WatchSource:0}: Error finding container 2a46af755e068eb55e1d5c7db4b1eff65e0154d830358bc85949926739965551: Status 404 returned error can't find the container with id 2a46af755e068eb55e1d5c7db4b1eff65e0154d830358bc85949926739965551
Oct 31 03:47:40.575767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.575727    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:47:40.584000 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:47:40.583893    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba63a01f_093e_4a6f_abb0_34c000a5b3ef.slice/crio-51957bab73926a96d477c87c637d629b22e52073b7b1da221c8f8f6151282f2d WatchSource:0}: Error finding container 51957bab73926a96d477c87c637d629b22e52073b7b1da221c8f8f6151282f2d: Status 404 returned error can't find the container with id 51957bab73926a96d477c87c637d629b22e52073b7b1da221c8f8f6151282f2d
Oct 31 03:47:40.587236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.587212    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:47:40.644286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.644253    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\") pod \"c402cc77-84ef-4713-9dc7-0f37a53ee135\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") "
Oct 31 03:47:40.644446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.644380    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\" (UniqueName: \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\") pod \"c402cc77-84ef-4713-9dc7-0f37a53ee135\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") "
Oct 31 03:47:40.644536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.644519    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\") pod \"c402cc77-84ef-4713-9dc7-0f37a53ee135\" (UID: \"c402cc77-84ef-4713-9dc7-0f37a53ee135\") "
Oct 31 03:47:40.644949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.644910    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca" pod "c402cc77-84ef-4713-9dc7-0f37a53ee135" (UID: "c402cc77-84ef-4713-9dc7-0f37a53ee135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:40.645388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.645363    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:40.647437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.647410    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca" pod "c402cc77-84ef-4713-9dc7-0f37a53ee135" (UID: "c402cc77-84ef-4713-9dc7-0f37a53ee135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:47:40.647542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.647486    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4" pod "c402cc77-84ef-4713-9dc7-0f37a53ee135" (UID: "c402cc77-84ef-4713-9dc7-0f37a53ee135"). InnerVolumeSpecName "kube-api-access-w7fq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:40.746802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.746766    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c402cc77-84ef-4713-9dc7-0f37a53ee135-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:40.746802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:40.746799    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7fq4\" (UniqueName: \"kubernetes.io/projected/c402cc77-84ef-4713-9dc7-0f37a53ee135-kube-api-access-w7fq4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:41.277373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.277335    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"
Oct 31 03:47:41.277957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.277373    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98" event={"ID":"c402cc77-84ef-4713-9dc7-0f37a53ee135","Type":"ContainerDied","Data":"7eb37302e49664bcb90fbb2e19fd937114eb9b1e12aa860c8f76209c5a4aad08"}
Oct 31 03:47:41.277957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.277431    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7eb37302e49664bcb90fbb2e19fd937114eb9b1e12aa860c8f76209c5a4aad08"
Oct 31 03:47:41.280599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.280566    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" event={"ID":"ba63a01f-093e-4a6f-abb0-34c000a5b3ef","Type":"ContainerStarted","Data":"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"}
Oct 31 03:47:41.280599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.280635    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" event={"ID":"ba63a01f-093e-4a6f-abb0-34c000a5b3ef","Type":"ContainerStarted","Data":"51957bab73926a96d477c87c637d629b22e52073b7b1da221c8f8f6151282f2d"}
Oct 31 03:47:41.280876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.280665    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:41.282532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.282500    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" event={"ID":"078cad6a-6136-451f-9da0-9747a8b7dac4","Type":"ContainerStarted","Data":"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"}
Oct 31 03:47:41.282685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.282552    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" event={"ID":"078cad6a-6136-451f-9da0-9747a8b7dac4","Type":"ContainerStarted","Data":"2a46af755e068eb55e1d5c7db4b1eff65e0154d830358bc85949926739965551"}
Oct 31 03:47:41.283046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.283024    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:41.284910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.284889    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:47:41.297893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.297833    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" podStartSLOduration=13.297820089 podStartE2EDuration="13.297820089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:41.296066463 +0000 UTC m=+1935.821286053" watchObservedRunningTime="2026-10-31 03:47:41.297820089 +0000 UTC m=+1935.823039677"
Oct 31 03:47:41.318404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.318349    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" podStartSLOduration=13.31833364 podStartE2EDuration="13.31833364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:41.312722458 +0000 UTC m=+1935.837942048" watchObservedRunningTime="2026-10-31 03:47:41.31833364 +0000 UTC m=+1935.843553240"
Oct 31 03:47:41.614863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:41.614771    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5df961-8bd7-46bb-8a90-1129a3840c56" path="/var/lib/kubelet/pods/2d5df961-8bd7-46bb-8a90-1129a3840c56/volumes"
Oct 31 03:47:51.854828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:51.853214    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:47:51.854828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:51.853484    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy" containerID="cri-o://c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76" gracePeriod=30
Oct 31 03:47:52.465453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:52.464906    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:47:52.473676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:52.467649    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:47:53.400019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.399980    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:47:53.400937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.400919    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c402cc77-84ef-4713-9dc7-0f37a53ee135" containerName="revoke-keys"
Oct 31 03:47:53.400989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.400938    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c402cc77-84ef-4713-9dc7-0f37a53ee135" containerName="revoke-keys"
Oct 31 03:47:53.401070 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.401061    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c402cc77-84ef-4713-9dc7-0f37a53ee135" containerName="revoke-keys"
Oct 31 03:47:53.405452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.405429    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.411094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.410812    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ce9b89\""
Oct 31 03:47:53.419062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.419032    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:47:53.666120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.666030    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9njm8\" (UniqueName: \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.666699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.666672    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.788630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.770989    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.788630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.771302    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9njm8\" (UniqueName: \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.800510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.797537    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:53.840430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:53.840393    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9njm8\" (UniqueName: \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\") pod \"payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:54.051504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.050130    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:54.473548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.473504    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:47:54.860574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.860539    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" event={"ID":"8636c4f5-1018-4f64-bf8c-7c7759097338","Type":"ContainerStarted","Data":"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"}
Oct 31 03:47:54.860817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.860619    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" event={"ID":"8636c4f5-1018-4f64-bf8c-7c7759097338","Type":"ContainerStarted","Data":"5f96bea67b84af039311d05e129b09689ae439899a93b1439ec937d9fe8a22d4"}
Oct 31 03:47:54.860817 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.860656    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:47:54.879743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:54.879687    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" podStartSLOduration=1.879669304 podStartE2EDuration="1.879669304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:54.87665813 +0000 UTC m=+1949.401877720" watchObservedRunningTime="2026-10-31 03:47:54.879669304 +0000 UTC m=+1949.404888892"
Oct 31 03:47:57.283623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.283578    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:47:57.328173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.328137    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:47:57.329218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.329198    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy"
Oct 31 03:47:57.329286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.329219    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy"
Oct 31 03:47:57.329379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.329370    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" containerName="istio-proxy"
Oct 31 03:47:57.340328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.340284    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.344000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.343957    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:47:57.347033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.347004    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ce9b89-dockercfg-p985b\""
Oct 31 03:47:57.381369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381334    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.381573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381389    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.381573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381437    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.381573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381465    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.381815 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381684    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.381878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381858    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.382000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.381980    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.382065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.382017    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.382129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.382110    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\") pod \"9331923e-a2df-4258-b289-9970c6cd3eda\" (UID: \"9331923e-a2df-4258-b289-9970c6cd3eda\") "
Oct 31 03:47:57.383038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.383009    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:57.383725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.383483    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:57.383725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.383669    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:57.383725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.383683    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:57.383965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.383777    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:47:57.385675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.385599    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:47:57.386125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.386100    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:57.386207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.386191    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "kube-api-access-n9dnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:47:57.386851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.386828    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo" pod "9331923e-a2df-4258-b289-9970c6cd3eda" (UID: "9331923e-a2df-4258-b289-9970c6cd3eda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:47:57.484835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.484793    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.485023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.484978    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfsqc\" (UniqueName: \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.485222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485204    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9331923e-a2df-4258-b289-9970c6cd3eda-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485226    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485241    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9dnt\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-kube-api-access-n9dnt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485257    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485272    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485286    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485301    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9331923e-a2df-4258-b289-9970c6cd3eda-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485319    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9331923e-a2df-4258-b289-9970c6cd3eda-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.485538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.485333    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9331923e-a2df-4258-b289-9970c6cd3eda-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:47:57.587261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.587218    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.587470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.587320    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfsqc\" (UniqueName: \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.588042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.588015    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.595762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.595729    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfsqc\" (UniqueName: \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\") pod \"payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.670558 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.670470    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:57.938088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.937992    5098 generic.go:356] "Generic (PLEG): container finished" podID="9331923e-a2df-4258-b289-9970c6cd3eda" containerID="c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76" exitCode=0
Oct 31 03:47:57.938269 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.938216    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" event={"ID":"9331923e-a2df-4258-b289-9970c6cd3eda","Type":"ContainerDied","Data":"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"}
Oct 31 03:47:57.938343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.938272    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w" event={"ID":"9331923e-a2df-4258-b289-9970c6cd3eda","Type":"ContainerDied","Data":"fcdfc81d48adc9d0407fa8f0c35d5a75091ac9ad361271b4149da05c95e79903"}
Oct 31 03:47:57.938343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.938302    5098 scope.go:122] "RemoveContainer" containerID="c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"
Oct 31 03:47:57.938502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.938485    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"
Oct 31 03:47:57.959532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.959504    5098 scope.go:122] "RemoveContainer" containerID="c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"
Oct 31 03:47:57.961123 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:47:57.961025    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76\": container with ID starting with c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76 not found: ID does not exist" containerID="c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"
Oct 31 03:47:57.961123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.961077    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76"} err="failed to get container status \"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76\": rpc error: code = NotFound desc = could not find container \"c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76\": container with ID starting with c1b6785b5bb9fe88f075e019d449c8610292e9287ad6561cd63904f5fd170a76 not found: ID does not exist"
Oct 31 03:47:57.971690 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.969191    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:47:57.978906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.978868    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:47:57.984916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:57.984880    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-fnq7w"]
Oct 31 03:47:59.028630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.025337    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" event={"ID":"77baa062-a718-4466-9847-1ea715d3e567","Type":"ContainerStarted","Data":"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"}
Oct 31 03:47:59.028630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.025408    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" event={"ID":"77baa062-a718-4466-9847-1ea715d3e567","Type":"ContainerStarted","Data":"517283fa51078e5247ba0f411cb7dad2a84fc2a839d967ded771da736872812a"}
Oct 31 03:47:59.035788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.031975    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:47:59.092922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.092838    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" podStartSLOduration=6.092815268 podStartE2EDuration="6.092815268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:47:59.085871805 +0000 UTC m=+1953.611091396" watchObservedRunningTime="2026-10-31 03:47:59.092815268 +0000 UTC m=+1953.618034858"
Oct 31 03:47:59.314365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.313272    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"]
Oct 31 03:47:59.401777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.398904    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"]
Oct 31 03:47:59.401777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.399046    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.405088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.405060    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:47:59.405411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.405395    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:47:59.530993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.530949    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.531184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.531000    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.531184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.531148    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhp2\" (UniqueName: \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.617869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.617781    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9331923e-a2df-4258-b289-9970c6cd3eda" path="/var/lib/kubelet/pods/9331923e-a2df-4258-b289-9970c6cd3eda/volumes"
Oct 31 03:47:59.633318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.633269    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.633534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.633329    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.633534 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.633391    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhp2\" (UniqueName: \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.634114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.634089    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.635936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.635916    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.642639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.642594    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhp2\" (UniqueName: \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\") pod \"maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:47:59.730881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:47:59.730844    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:48:00.028322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:00.028291    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"]
Oct 31 03:48:00.033457 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:48:00.033408    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c188efb_1813_4ffb_b10b_b238aac28444.slice/crio-711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b WatchSource:0}: Error finding container 711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b: Status 404 returned error can't find the container with id 711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b
Oct 31 03:48:00.087368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:00.087327    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq" event={"ID":"7c188efb-1813-4ffb-b10b-b238aac28444","Type":"ContainerStarted","Data":"711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b"}
Oct 31 03:48:01.065207 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:01.064816    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:48:01.091909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:01.091874    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq" event={"ID":"7c188efb-1813-4ffb-b10b-b238aac28444","Type":"ContainerStarted","Data":"0037e028b55f834701ae3357fb02918de2038e99adbb7272722c20a8b3f6bb66"}
Oct 31 03:48:01.111999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:01.111945    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq" podStartSLOduration=2.111930242 podStartE2EDuration="2.111930242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:01.110248272 +0000 UTC m=+1955.635467864" watchObservedRunningTime="2026-10-31 03:48:01.111930242 +0000 UTC m=+1955.637149832"
Oct 31 03:48:02.103106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:02.103066    5098 generic.go:356] "Generic (PLEG): container finished" podID="7c188efb-1813-4ffb-b10b-b238aac28444" containerID="0037e028b55f834701ae3357fb02918de2038e99adbb7272722c20a8b3f6bb66" exitCode=0
Oct 31 03:48:02.103498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:02.103149    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq" event={"ID":"7c188efb-1813-4ffb-b10b-b238aac28444","Type":"ContainerDied","Data":"0037e028b55f834701ae3357fb02918de2038e99adbb7272722c20a8b3f6bb66"}
Oct 31 03:48:03.324765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.324734    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:48:03.448116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.448033    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\" (UniqueName: \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\") pod \"7c188efb-1813-4ffb-b10b-b238aac28444\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") "
Oct 31 03:48:03.448279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.448202    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\") pod \"7c188efb-1813-4ffb-b10b-b238aac28444\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") "
Oct 31 03:48:03.448400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.448377    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\") pod \"7c188efb-1813-4ffb-b10b-b238aac28444\" (UID: \"7c188efb-1813-4ffb-b10b-b238aac28444\") "
Oct 31 03:48:03.448601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.448570    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca" pod "7c188efb-1813-4ffb-b10b-b238aac28444" (UID: "7c188efb-1813-4ffb-b10b-b238aac28444"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:03.449204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.449181    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.450659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.450592    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca" pod "7c188efb-1813-4ffb-b10b-b238aac28444" (UID: "7c188efb-1813-4ffb-b10b-b238aac28444"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:03.450770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.450725    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2" pod "7c188efb-1813-4ffb-b10b-b238aac28444" (UID: "7c188efb-1813-4ffb-b10b-b238aac28444"). InnerVolumeSpecName "kube-api-access-thhp2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:03.550626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.550573    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thhp2\" (UniqueName: \"kubernetes.io/projected/7c188efb-1813-4ffb-b10b-b238aac28444-kube-api-access-thhp2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:03.550626 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:03.550621    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c188efb-1813-4ffb-b10b-b238aac28444-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:04.122262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:04.122229    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"
Oct 31 03:48:04.122262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:04.122249    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq" event={"ID":"7c188efb-1813-4ffb-b10b-b238aac28444","Type":"ContainerDied","Data":"711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b"}
Oct 31 03:48:04.122536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:04.122300    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="711dfd64c7122ab53e0b701ef1d263ba7b20b77405d95d6c10a81cb29fdf536b"
Oct 31 03:48:05.880358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:05.880328    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:48:06.602126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:06.602085    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:48:06.602333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:06.602307    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy" containerID="cri-o://26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0" gracePeriod=30
Oct 31 03:48:07.413377 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:07.413328    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:48:07.414570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:07.414533    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy" containerID="cri-o://35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c" gracePeriod=30
Oct 31 03:48:08.240632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:08.235491    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:48:08.240632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:08.235749    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" podUID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" containerName="maas-api" containerID="cri-o://aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244" gracePeriod=30
Oct 31 03:48:08.719628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:08.715668    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"]
Oct 31 03:48:08.736631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:08.733260    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c5dcb7-vrf98"]
Oct 31 03:48:09.148399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.148369    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:48:09.240802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.240766    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\" (UniqueName: \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\") pod \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") "
Oct 31 03:48:09.241024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.240964    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\") pod \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") "
Oct 31 03:48:09.241024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.241013    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\") pod \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") "
Oct 31 03:48:09.241180 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.241057    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\") pod \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\" (UID: \"55b0fb44-1dac-45dd-aae3-7919258ff8a6\") "
Oct 31 03:48:09.242114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.242064    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca" pod "55b0fb44-1dac-45dd-aae3-7919258ff8a6" (UID: "55b0fb44-1dac-45dd-aae3-7919258ff8a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:09.243429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.243398    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw" pod "55b0fb44-1dac-45dd-aae3-7919258ff8a6" (UID: "55b0fb44-1dac-45dd-aae3-7919258ff8a6"). InnerVolumeSpecName "kube-api-access-qwrbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:09.243787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.243760    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs" pod "55b0fb44-1dac-45dd-aae3-7919258ff8a6" (UID: "55b0fb44-1dac-45dd-aae3-7919258ff8a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:09.244014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.243988    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls" pod "55b0fb44-1dac-45dd-aae3-7919258ff8a6" (UID: "55b0fb44-1dac-45dd-aae3-7919258ff8a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:09.314735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.314697    5098 generic.go:356] "Generic (PLEG): container finished" podID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" containerID="aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244" exitCode=0
Oct 31 03:48:09.314949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.314753    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"
Oct 31 03:48:09.314949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.314776    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" event={"ID":"55b0fb44-1dac-45dd-aae3-7919258ff8a6","Type":"ContainerDied","Data":"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"}
Oct 31 03:48:09.314949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.314872    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn" event={"ID":"55b0fb44-1dac-45dd-aae3-7919258ff8a6","Type":"ContainerDied","Data":"2a332b97f24d12337e9b67b0f4567cd0978a8ab80875cf360a4bb024b885680a"}
Oct 31 03:48:09.314949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.314900    5098 scope.go:122] "RemoveContainer" containerID="aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"
Oct 31 03:48:09.329733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.329713    5098 scope.go:122] "RemoveContainer" containerID="aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"
Oct 31 03:48:09.330056 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:09.330034    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244\": container with ID starting with aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244 not found: ID does not exist" containerID="aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"
Oct 31 03:48:09.330135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.330065    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244"} err="failed to get container status \"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244\": rpc error: code = NotFound desc = could not find container \"aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244\": container with ID starting with aca845e267621f382031693b513660215367bd9c0c67a97e698955e08a097244 not found: ID does not exist"
Oct 31 03:48:09.343900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.343865    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:09.343900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.343891    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55b0fb44-1dac-45dd-aae3-7919258ff8a6-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:09.343900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.343904    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/55b0fb44-1dac-45dd-aae3-7919258ff8a6-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:09.344159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.343914    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrbw\" (UniqueName: \"kubernetes.io/projected/55b0fb44-1dac-45dd-aae3-7919258ff8a6-kube-api-access-qwrbw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:09.344159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.344113    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:48:09.349385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.349357    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c5dcb7-6db588c4cd-pr4sn"]
Oct 31 03:48:09.616352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.616309    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" path="/var/lib/kubelet/pods/55b0fb44-1dac-45dd-aae3-7919258ff8a6/volumes"
Oct 31 03:48:09.616996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:09.616977    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c402cc77-84ef-4713-9dc7-0f37a53ee135" path="/var/lib/kubelet/pods/c402cc77-84ef-4713-9dc7-0f37a53ee135/volumes"
Oct 31 03:48:11.096118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:11.096075    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:48:11.928290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:11.928262    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:48:12.030590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030534    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.030590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030591    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.030904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030745    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.030904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030798    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.030904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030854    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.030904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030880    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.030904 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030902    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.031200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030887    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.031200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030938    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.031200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.030967    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.031200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031024    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\") pod \"eafcb788-673c-4aaa-8b58-2e6defe730b1\" (UID: \"eafcb788-673c-4aaa-8b58-2e6defe730b1\") "
Oct 31 03:48:12.031200 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031112    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.031466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031303    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.031466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031322    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:12.031991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031963    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eafcb788-673c-4aaa-8b58-2e6defe730b1-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.032105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.031994    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.032105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.032009    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.032105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.032023    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.032105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.032037    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.034461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.034423    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.034839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.034809    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "kube-api-access-c26nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:12.034985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.034960    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:12.035755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.035731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo" pod "eafcb788-673c-4aaa-8b58-2e6defe730b1" (UID: "eafcb788-673c-4aaa-8b58-2e6defe730b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:12.161682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.161526    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.161682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.161569    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.161682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.161585    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c26nm\" (UniqueName: \"kubernetes.io/projected/eafcb788-673c-4aaa-8b58-2e6defe730b1-kube-api-access-c26nm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.161682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.161621    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eafcb788-673c-4aaa-8b58-2e6defe730b1-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.369171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.365802    5098 generic.go:356] "Generic (PLEG): container finished" podID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerID="26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0" exitCode=0
Oct 31 03:48:12.369171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.366324    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" event={"ID":"eafcb788-673c-4aaa-8b58-2e6defe730b1","Type":"ContainerDied","Data":"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"}
Oct 31 03:48:12.369171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.366426    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg" event={"ID":"eafcb788-673c-4aaa-8b58-2e6defe730b1","Type":"ContainerDied","Data":"9c31c466e0eeed54118200ba35e48a327d8a3da75053000acadd86e418ef2295"}
Oct 31 03:48:12.369171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.366475    5098 scope.go:122] "RemoveContainer" containerID="26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"
Oct 31 03:48:12.369171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.366752    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"
Oct 31 03:48:12.386512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.386486    5098 scope.go:122] "RemoveContainer" containerID="26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"
Oct 31 03:48:12.386918 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:12.386889    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0\": container with ID starting with 26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0 not found: ID does not exist" containerID="26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"
Oct 31 03:48:12.386997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.386936    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0"} err="failed to get container status \"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0\": rpc error: code = NotFound desc = could not find container \"26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0\": container with ID starting with 26bb0659f0a8011c9393f26a282b2daa5330edd418ac08197abe50c0d54766f0 not found: ID does not exist"
Oct 31 03:48:12.405258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.405223    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:48:12.410473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.410441    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-pt5jg"]
Oct 31 03:48:12.747717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.747680    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:48:12.882866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.882370    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.883073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.882925    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.883496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.883467    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.883838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.883810    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.884038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.883955    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.884050    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.884141    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.884315    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.884419    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.884477    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\") pod \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\" (UID: \"ee2dd46e-8261-4689-a7db-a708bf76fc0e\") "
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.885547    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.886059    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.886901    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.887190    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.890984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.887400    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.938633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.934826    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "kube-api-access-dvjk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:12.941340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.940744    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:12.944418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.944383    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:12.944949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.944903    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token" pod "ee2dd46e-8261-4689-a7db-a708bf76fc0e" (UID: "ee2dd46e-8261-4689-a7db-a708bf76fc0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998120    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998161    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998173    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998186    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvjk2\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-kube-api-access-dvjk2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998201    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998212    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998223    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:12.998705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:12.998234    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee2dd46e-8261-4689-a7db-a708bf76fc0e-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:13.416841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.416809    5098 generic.go:356] "Generic (PLEG): container finished" podID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerID="35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c" exitCode=0
Oct 31 03:48:13.417381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.417361    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" event={"ID":"ee2dd46e-8261-4689-a7db-a708bf76fc0e","Type":"ContainerDied","Data":"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"}
Oct 31 03:48:13.417539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.417523    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j" event={"ID":"ee2dd46e-8261-4689-a7db-a708bf76fc0e","Type":"ContainerDied","Data":"e01f4914b261a9442a518a880824d1602acc5aa707d146503017351e24e2a13b"}
Oct 31 03:48:13.417865 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.417846    5098 scope.go:122] "RemoveContainer" containerID="35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"
Oct 31 03:48:13.418110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.418093    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"
Oct 31 03:48:13.442096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.442064    5098 scope.go:122] "RemoveContainer" containerID="35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"
Oct 31 03:48:13.442494 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:13.442472    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c\": container with ID starting with 35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c not found: ID does not exist" containerID="35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"
Oct 31 03:48:13.442559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.442506    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c"} err="failed to get container status \"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c\": rpc error: code = NotFound desc = could not find container \"35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c\": container with ID starting with 35d6a784234c2a868feb3b6197696b3d02486de2528488468d0a3146fb9a094c not found: ID does not exist"
Oct 31 03:48:13.458840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.458797    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:48:13.464004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.463973    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-cnn4j"]
Oct 31 03:48:13.614517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.614478    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" path="/var/lib/kubelet/pods/eafcb788-673c-4aaa-8b58-2e6defe730b1/volumes"
Oct 31 03:48:13.615224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:13.615198    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" path="/var/lib/kubelet/pods/ee2dd46e-8261-4689-a7db-a708bf76fc0e/volumes"
Oct 31 03:48:15.678186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:15.678154    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:48:15.678719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:15.678369    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" containerID="cri-o://064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a" gracePeriod=30
Oct 31 03:48:16.194093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.194044    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"]
Oct 31 03:48:16.195300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195277    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy"
Oct 31 03:48:16.195300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195301    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195318    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7c188efb-1813-4ffb-b10b-b238aac28444" containerName="revoke-keys"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195323    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c188efb-1813-4ffb-b10b-b238aac28444" containerName="revoke-keys"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195363    5098 cpu_manager.go:414] "containerMap: removing container" podUID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" containerName="maas-api"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195369    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" containerName="maas-api"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195377    5098 cpu_manager.go:414] "containerMap: removing container" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy"
Oct 31 03:48:16.195454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195381    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy"
Oct 31 03:48:16.195742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195556    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b0fb44-1dac-45dd-aae3-7919258ff8a6" containerName="maas-api"
Oct 31 03:48:16.195742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195568    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="eafcb788-673c-4aaa-8b58-2e6defe730b1" containerName="istio-proxy"
Oct 31 03:48:16.195742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195594    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee2dd46e-8261-4689-a7db-a708bf76fc0e" containerName="istio-proxy"
Oct 31 03:48:16.195742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.195620    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c188efb-1813-4ffb-b10b-b238aac28444" containerName="revoke-keys"
Oct 31 03:48:16.201816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.201789    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.208275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.208236    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"]
Oct 31 03:48:16.249552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.249504    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:48:16.249769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.249571    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:48:16.263844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.263806    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.264095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.263860    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2lfg\" (UniqueName: \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.264095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.264007    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.366266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.366221    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.366499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.366286    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2lfg\" (UniqueName: \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.366499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.366360    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.367073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.367046    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.368838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.368817    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.374778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.374757    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2lfg\" (UniqueName: \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\") pod \"maas-api-revoke-keys-e2e-inf-b-73e222-xpg84\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.530163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.530124    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:16.824978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:16.824939    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"]
Oct 31 03:48:16.826662 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:48:16.826582    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca4b5f3_c169_40b8_8639_d4d32efbc6b5.slice/crio-4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703 WatchSource:0}: Error finding container 4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703: Status 404 returned error can't find the container with id 4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703
Oct 31 03:48:17.480893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:17.480807    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84" event={"ID":"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5","Type":"ContainerStarted","Data":"107b6483d5deb90bafb269762fcc2b04d697e903dafbec0334b43c4ace2178c1"}
Oct 31 03:48:17.480893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:17.480866    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84" event={"ID":"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5","Type":"ContainerStarted","Data":"4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703"}
Oct 31 03:48:17.537971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:17.537893    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84" podStartSLOduration=1.537869192 podStartE2EDuration="1.537869192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:17.50619611 +0000 UTC m=+1972.031415703" watchObservedRunningTime="2026-10-31 03:48:17.537869192 +0000 UTC m=+1972.063088782"
Oct 31 03:48:18.549515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:18.549475    5098 generic.go:356] "Generic (PLEG): container finished" podID="7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" containerID="107b6483d5deb90bafb269762fcc2b04d697e903dafbec0334b43c4ace2178c1" exitCode=0
Oct 31 03:48:18.561150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:18.559306    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84" event={"ID":"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5","Type":"ContainerDied","Data":"107b6483d5deb90bafb269762fcc2b04d697e903dafbec0334b43c4ace2178c1"}
Oct 31 03:48:19.158433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.158393    5098 patch_prober.go:29] interesting pod/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:48:19.158636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.158448    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:48:19.767887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.767854    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:19.909683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.903214    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\") pod \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") "
Oct 31 03:48:19.909683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.903694    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\") pod \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") "
Oct 31 03:48:19.909683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.903777    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\" (UniqueName: \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\") pod \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\" (UID: \"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5\") "
Oct 31 03:48:19.917753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.911816    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg" pod "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" (UID: "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5"). InnerVolumeSpecName "kube-api-access-m2lfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:19.944891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.942966    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca" pod "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" (UID: "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:19.950529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:19.950345    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca" pod "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" (UID: "7ca4b5f3-c169-40b8-8639-d4d32efbc6b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:20.006535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.006493    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:20.006535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.006535    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2lfg\" (UniqueName: \"kubernetes.io/projected/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-kube-api-access-m2lfg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:20.006841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.006549    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:20.585074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.585036    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84" event={"ID":"7ca4b5f3-c169-40b8-8639-d4d32efbc6b5","Type":"ContainerDied","Data":"4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703"}
Oct 31 03:48:20.585279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.585094    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d0fd2df95b88349605e1cd31dee8bd2371ab953e76bac7c6226f04bff9a3703"
Oct 31 03:48:20.585279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.585093    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"
Oct 31 03:48:20.982065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:20.982033    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:48:21.133667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133520    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.133667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133570    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.133667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133643    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133778    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133844    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133876    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133905    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.133926    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134015    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134041    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134061    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134146    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\") pod \"138d3b3c-cada-4754-9f08-25a848c726a5\" (UID: \"138d3b3c-cada-4754-9f08-25a848c726a5\") "
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134314    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:21.134552 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.134392    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:21.135169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.135148    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.135264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.135174    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.135264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.135189    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.135264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.135203    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.135264 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.135217    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138d3b3c-cada-4754-9f08-25a848c726a5-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.136342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.136318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:21.136448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.136374    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "kube-api-access-8tsh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:21.136514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.136496    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:21.136748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.136731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token" pod "138d3b3c-cada-4754-9f08-25a848c726a5" (UID: "138d3b3c-cada-4754-9f08-25a848c726a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:21.241647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.241485    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138d3b3c-cada-4754-9f08-25a848c726a5-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.241647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.241536    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138d3b3c-cada-4754-9f08-25a848c726a5-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.241647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.241552    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsh7\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-kube-api-access-8tsh7\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.241647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.241572    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138d3b3c-cada-4754-9f08-25a848c726a5-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:21.596279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.596244    5098 generic.go:356] "Generic (PLEG): container finished" podID="138d3b3c-cada-4754-9f08-25a848c726a5" containerID="064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a" exitCode=0
Oct 31 03:48:21.596452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.596299    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"
Oct 31 03:48:21.596452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.596339    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerDied","Data":"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"}
Oct 31 03:48:21.596452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.596404    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj" event={"ID":"138d3b3c-cada-4754-9f08-25a848c726a5","Type":"ContainerDied","Data":"91c510a5182bb560c56daecfa08ad141f8083e3ccd32a2eb790731b8ec6cdead"}
Oct 31 03:48:21.596452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.596435    5098 scope.go:122] "RemoveContainer" containerID="064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"
Oct 31 03:48:21.610679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.610659    5098 scope.go:122] "RemoveContainer" containerID="65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"
Oct 31 03:48:21.625176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.625153    5098 scope.go:122] "RemoveContainer" containerID="064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"
Oct 31 03:48:21.625527 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:21.625507    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a\": container with ID starting with 064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a not found: ID does not exist" containerID="064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"
Oct 31 03:48:21.625576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.625536    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a"} err="failed to get container status \"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a\": rpc error: code = NotFound desc = could not find container \"064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a\": container with ID starting with 064d985b3ef8f20ca8e360e06e5fc0eba0ec8cda646602ba75417c223ccc3c9a not found: ID does not exist"
Oct 31 03:48:21.625576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.625554    5098 scope.go:122] "RemoveContainer" containerID="65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"
Oct 31 03:48:21.625867 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:21.625834    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485\": container with ID starting with 65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485 not found: ID does not exist" containerID="65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"
Oct 31 03:48:21.625918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.625869    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485"} err="failed to get container status \"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485\": rpc error: code = NotFound desc = could not find container \"65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485\": container with ID starting with 65b476ac0c6e0861ada9aaf46812b4c594c58d2941b9e2575410970a11086485 not found: ID does not exist"
Oct 31 03:48:21.627300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.627273    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:48:21.632517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:21.632489    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c5dcb7-openshift-default-6686fcb7bc-mpmpj"]
Oct 31 03:48:23.628989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.628950    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" path="/var/lib/kubelet/pods/138d3b3c-cada-4754-9f08-25a848c726a5/volumes"
Oct 31 03:48:23.855079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.855026    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:48:23.855990 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.855971    5098 cpu_manager.go:414] "containerMap: removing container" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.855991    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856005    5098 cpu_manager.go:414] "containerMap: removing container" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856010    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856040    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" containerName="revoke-keys"
Oct 31 03:48:23.856052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856046    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" containerName="revoke-keys"
Oct 31 03:48:23.856222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856179    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856189    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="138d3b3c-cada-4754-9f08-25a848c726a5" containerName="istio-proxy"
Oct 31 03:48:23.856222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.856199    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" containerName="revoke-keys"
Oct 31 03:48:23.861927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.861892    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:23.868158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.868127    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-63258b\""
Oct 31 03:48:23.869018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.868853    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:48:23.883999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.883916    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:48:23.890645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.890617    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:23.898964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.898939    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:48:23.912947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.912918    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-63258b-dockercfg-t9n6m\""
Oct 31 03:48:23.941234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.941189    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:23.941457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.941402    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjd29\" (UniqueName: \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:23.941525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.941506    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmmbn\" (UniqueName: \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:23.941582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:23.941550    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.077631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.070966    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjd29\" (UniqueName: \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:24.077631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.071072    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmmbn\" (UniqueName: \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.077631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.071146    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.077631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.071478    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:24.104631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.091540    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:24.104631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.092396    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.124631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.118738    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmmbn\" (UniqueName: \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\") pod \"payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.143774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.137464    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjd29\" (UniqueName: \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\") pod \"payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:24.184085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.182867    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:24.231231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.229997    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:24.645419 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:48:24.645386    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod528eb2c5_8091_4f0c_b8a5_cb98a68e255f.slice/crio-1a37ceeec16027ec4eb5173835710177f84f654c5c7e605fb237aba205281c7a WatchSource:0}: Error finding container 1a37ceeec16027ec4eb5173835710177f84f654c5c7e605fb237aba205281c7a: Status 404 returned error can't find the container with id 1a37ceeec16027ec4eb5173835710177f84f654c5c7e605fb237aba205281c7a
Oct 31 03:48:24.646132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.646108    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:48:24.792413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:24.792381    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:48:24.793859 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:48:24.793828    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1bfcaa4_9246_4137_b4d4_83ab590f40b8.slice/crio-830f6185203fb8b4f3e28cf71b96ac9d354c1873bd1df6c72e52e664e34ae90d WatchSource:0}: Error finding container 830f6185203fb8b4f3e28cf71b96ac9d354c1873bd1df6c72e52e664e34ae90d: Status 404 returned error can't find the container with id 830f6185203fb8b4f3e28cf71b96ac9d354c1873bd1df6c72e52e664e34ae90d
Oct 31 03:48:25.225881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.225843    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:48:25.226092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.226069    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" podUID="4bff431f-28d7-43a6-9181-e7333aceb93e" containerName="maas-api" containerID="cri-o://7329dabf3a0035affe5b28fa28cbffb68cb0a1ee3836906970c3a7e1ba3115a3" gracePeriod=30
Oct 31 03:48:25.471473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.471433    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"]
Oct 31 03:48:25.489076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.488955    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-880e04-6nwxq"]
Oct 31 03:48:25.934756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.934713    5098 generic.go:356] "Generic (PLEG): container finished" podID="4bff431f-28d7-43a6-9181-e7333aceb93e" containerID="7329dabf3a0035affe5b28fa28cbffb68cb0a1ee3836906970c3a7e1ba3115a3" exitCode=0
Oct 31 03:48:25.939265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.938551    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c188efb-1813-4ffb-b10b-b238aac28444" path="/var/lib/kubelet/pods/7c188efb-1813-4ffb-b10b-b238aac28444/volumes"
Oct 31 03:48:25.939265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.939117    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" event={"ID":"4bff431f-28d7-43a6-9181-e7333aceb93e","Type":"ContainerDied","Data":"7329dabf3a0035affe5b28fa28cbffb68cb0a1ee3836906970c3a7e1ba3115a3"}
Oct 31 03:48:25.972292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.966896    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" event={"ID":"528eb2c5-8091-4f0c-b8a5-cb98a68e255f","Type":"ContainerStarted","Data":"d580e2387365df950472fc9b295e8d9a777a7ab50d26912115e6c3d38233cae3"}
Oct 31 03:48:25.972292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.966966    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" event={"ID":"528eb2c5-8091-4f0c-b8a5-cb98a68e255f","Type":"ContainerStarted","Data":"1a37ceeec16027ec4eb5173835710177f84f654c5c7e605fb237aba205281c7a"}
Oct 31 03:48:25.972292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.967501    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:26.000729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.999705    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" event={"ID":"b1bfcaa4-9246-4137-b4d4-83ab590f40b8","Type":"ContainerStarted","Data":"9ed2396eb4e98102635a9ea166154c5f18ed41db1b864a3bfa7d98f47aca9521"}
Oct 31 03:48:26.000729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.999772    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" event={"ID":"b1bfcaa4-9246-4137-b4d4-83ab590f40b8","Type":"ContainerStarted","Data":"830f6185203fb8b4f3e28cf71b96ac9d354c1873bd1df6c72e52e664e34ae90d"}
Oct 31 03:48:26.000729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:25.999949    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:26.004932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.003928    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:48:26.009192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.009092    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" podStartSLOduration=3.009071999 podStartE2EDuration="3.009071999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:26.001831579 +0000 UTC m=+1980.527051168" watchObservedRunningTime="2026-10-31 03:48:26.009071999 +0000 UTC m=+1980.534291591"
Oct 31 03:48:26.026955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.026904    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" podStartSLOduration=3.026887946 podStartE2EDuration="3.026887946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:26.021040131 +0000 UTC m=+1980.546259722" watchObservedRunningTime="2026-10-31 03:48:26.026887946 +0000 UTC m=+1980.552107567"
Oct 31 03:48:26.232421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.232321    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\") pod \"4bff431f-28d7-43a6-9181-e7333aceb93e\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") "
Oct 31 03:48:26.232773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.232752    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\") pod \"4bff431f-28d7-43a6-9181-e7333aceb93e\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") "
Oct 31 03:48:26.232978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.232962    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\" (UniqueName: \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\") pod \"4bff431f-28d7-43a6-9181-e7333aceb93e\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") "
Oct 31 03:48:26.233220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.233199    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\") pod \"4bff431f-28d7-43a6-9181-e7333aceb93e\" (UID: \"4bff431f-28d7-43a6-9181-e7333aceb93e\") "
Oct 31 03:48:26.237761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.237714    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca" pod "4bff431f-28d7-43a6-9181-e7333aceb93e" (UID: "4bff431f-28d7-43a6-9181-e7333aceb93e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:26.251446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.251399    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg" pod "4bff431f-28d7-43a6-9181-e7333aceb93e" (UID: "4bff431f-28d7-43a6-9181-e7333aceb93e"). InnerVolumeSpecName "kube-api-access-gmbhg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:26.267074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.254679    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs" pod "4bff431f-28d7-43a6-9181-e7333aceb93e" (UID: "4bff431f-28d7-43a6-9181-e7333aceb93e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:26.267074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.256493    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls" pod "4bff431f-28d7-43a6-9181-e7333aceb93e" (UID: "4bff431f-28d7-43a6-9181-e7333aceb93e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:26.335926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.335891    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:26.335926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.335924    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4bff431f-28d7-43a6-9181-e7333aceb93e-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:26.336138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.335964    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmbhg\" (UniqueName: \"kubernetes.io/projected/4bff431f-28d7-43a6-9181-e7333aceb93e-kube-api-access-gmbhg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:26.336138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:26.335976    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bff431f-28d7-43a6-9181-e7333aceb93e-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:27.059762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.059721    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz" event={"ID":"4bff431f-28d7-43a6-9181-e7333aceb93e","Type":"ContainerDied","Data":"d7432eb2651f931b987a3355a64cd64a582e476b5557f33d029e4b303dc56822"}
Oct 31 03:48:27.060310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.060280    5098 scope.go:122] "RemoveContainer" containerID="7329dabf3a0035affe5b28fa28cbffb68cb0a1ee3836906970c3a7e1ba3115a3"
Oct 31 03:48:27.060496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.060479    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"
Oct 31 03:48:27.102340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.102295    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:48:27.108282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.108248    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-880e04-67cf98484-ntxfz"]
Oct 31 03:48:27.616003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:27.615966    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bff431f-28d7-43a6-9181-e7333aceb93e" path="/var/lib/kubelet/pods/4bff431f-28d7-43a6-9181-e7333aceb93e/volumes"
Oct 31 03:48:31.116334 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:31.116295    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:48:37.063805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:37.063775    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:48:37.064364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:37.064180    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:48:39.029020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:39.028986    5098 scope.go:122] "RemoveContainer" containerID="2b8b9f39e2f54d3211eec0a65733c1c371d7bf20722af3941996ee2de934a2f8"
Oct 31 03:48:41.901110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.901073    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:48:41.902551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.902521    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4bff431f-28d7-43a6-9181-e7333aceb93e" containerName="maas-api"
Oct 31 03:48:41.902551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.902551    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bff431f-28d7-43a6-9181-e7333aceb93e" containerName="maas-api"
Oct 31 03:48:41.902800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.902784    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bff431f-28d7-43a6-9181-e7333aceb93e" containerName="maas-api"
Oct 31 03:48:41.910332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.910268    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:41.919287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.919253    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-63258b-serving-cert\""
Oct 31 03:48:41.926433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:41.926402    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:48:42.017986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.017949    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.051667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.051619    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57pcq\" (UniqueName: \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.057915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.057876    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.076054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.076013    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.178451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.178353    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.178649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.178455    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57pcq\" (UniqueName: \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.178649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.178483    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.178649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.178566    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.179354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.179326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.181100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.181074    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.181209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.181143    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.186893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.186864    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57pcq\" (UniqueName: \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\") pod \"maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.242128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.242095    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:42.645861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.645825    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:48:42.646134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.646104    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" podUID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" containerName="maas-api" containerID="cri-o://337cb3fb34e4848cdc7a89ad263d628aa38f162a51e2a5b21b5ebbd842336763" gracePeriod=30
Oct 31 03:48:42.660473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.660435    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:48:42.661601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.661264    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:48:42.967682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:42.960343    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"]
Oct 31 03:48:43.051993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.051919    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-73e222-xpg84"]
Oct 31 03:48:43.311211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.311177    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" event={"ID":"07c8d566-0a7f-43b7-bb17-cfe678e2ebae","Type":"ContainerStarted","Data":"44f4901b21f344d562f517e6919ab53edf164ae49b7336ceadbf8d292fecad86"}
Oct 31 03:48:43.327246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.327004    5098 generic.go:356] "Generic (PLEG): container finished" podID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" containerID="337cb3fb34e4848cdc7a89ad263d628aa38f162a51e2a5b21b5ebbd842336763" exitCode=0
Oct 31 03:48:43.327246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.327080    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" event={"ID":"ad699c26-7f19-4bcb-8505-397e23a3ceb5","Type":"ContainerDied","Data":"337cb3fb34e4848cdc7a89ad263d628aa38f162a51e2a5b21b5ebbd842336763"}
Oct 31 03:48:43.514665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.514633    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:48:43.632466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.632366    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca4b5f3-c169-40b8-8639-d4d32efbc6b5" path="/var/lib/kubelet/pods/7ca4b5f3-c169-40b8-8639-d4d32efbc6b5/volumes"
Oct 31 03:48:43.687956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.687918    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\") pod \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") "
Oct 31 03:48:43.688188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.688055    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\" (UniqueName: \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\") pod \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") "
Oct 31 03:48:43.688188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.688097    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\") pod \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") "
Oct 31 03:48:43.688341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.688241    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\") pod \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\" (UID: \"ad699c26-7f19-4bcb-8505-397e23a3ceb5\") "
Oct 31 03:48:43.688660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.688623    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca" pod "ad699c26-7f19-4bcb-8505-397e23a3ceb5" (UID: "ad699c26-7f19-4bcb-8505-397e23a3ceb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:43.689056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.689036    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad699c26-7f19-4bcb-8505-397e23a3ceb5-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:43.690353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.690326    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj" pod "ad699c26-7f19-4bcb-8505-397e23a3ceb5" (UID: "ad699c26-7f19-4bcb-8505-397e23a3ceb5"). InnerVolumeSpecName "kube-api-access-275lj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:43.690446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.690364    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls" pod "ad699c26-7f19-4bcb-8505-397e23a3ceb5" (UID: "ad699c26-7f19-4bcb-8505-397e23a3ceb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:43.690446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.690377    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs" pod "ad699c26-7f19-4bcb-8505-397e23a3ceb5" (UID: "ad699c26-7f19-4bcb-8505-397e23a3ceb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:48:43.791116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.791076    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:43.791116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.791114    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-275lj\" (UniqueName: \"kubernetes.io/projected/ad699c26-7f19-4bcb-8505-397e23a3ceb5-kube-api-access-275lj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:43.791116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:43.791126    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ad699c26-7f19-4bcb-8505-397e23a3ceb5-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:44.337126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.337098    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"
Oct 31 03:48:44.337672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.337097    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb" event={"ID":"ad699c26-7f19-4bcb-8505-397e23a3ceb5","Type":"ContainerDied","Data":"34df790dbba8cc54dc3492d55cc5bbf8365b207901a689e5c89aa250be69033f"}
Oct 31 03:48:44.337672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.337256    5098 scope.go:122] "RemoveContainer" containerID="337cb3fb34e4848cdc7a89ad263d628aa38f162a51e2a5b21b5ebbd842336763"
Oct 31 03:48:44.339152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.339129    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" event={"ID":"07c8d566-0a7f-43b7-bb17-cfe678e2ebae","Type":"ContainerStarted","Data":"60b4c6eb6512954fd98425d854723a838dce877c6f0cbd11fd0a5e619b2901d3"}
Oct 31 03:48:44.339259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.339195    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:44.365118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.365068    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" podStartSLOduration=2.725049948 podStartE2EDuration="3.365054526s" totalImagesPullingTime="640.004578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:48:44.36252651 +0000 UTC m=+1998.887746121" watchObservedRunningTime="2026-10-31 03:48:44.365054526 +0000 UTC m=+1998.890274115"
Oct 31 03:48:44.386504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.386474    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:48:44.391527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:44.391499    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-73e222-5cdbd9d966-hpqgb"]
Oct 31 03:48:45.616932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:45.616896    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" path="/var/lib/kubelet/pods/ad699c26-7f19-4bcb-8505-397e23a3ceb5/volumes"
Oct 31 03:48:49.875906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:49.875871    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:48:49.876489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:49.876088    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" containerID="cri-o://e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d" gracePeriod=30
Oct 31 03:48:50.353965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:50.353936    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:48:50.893993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:50.893955    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:48:52.699635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:52.689906    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:48:52.699635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:52.690181    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy" containerID="cri-o://b08440013915e9d6100ab2dab5dc8de865394a14fe2ee7b37f4f6536a1416a4e" gracePeriod=30
Oct 31 03:48:54.291076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:54.291033    5098 patch_prober.go:29] interesting pod/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:48:54.291513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:54.291115    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:48:55.189858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.189823    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:48:55.232220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232177    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232300    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232465    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232576 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232510    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232664    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232667    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.232762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232719    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232816    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.232882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232875    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.233001 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232953    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\") pod \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\" (UID: \"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a\") "
Oct 31 03:48:55.233684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232741    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.233684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.232964    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.233684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233137    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.233684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233493    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:55.234051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233791    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.234051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233813    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.234051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233826    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.234051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233840    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.234051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.233854    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.234863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.234833    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:55.236029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.236000    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:55.236143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.236085    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:55.236216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.236194    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9" pod "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" (UID: "5bd9e3a9-0f94-493f-9d4c-18e6302ae53a"). InnerVolumeSpecName "kube-api-access-bfgz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:55.335517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.335483    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.335517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.335514    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.335517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.335524    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.336058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.335535    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfgz9\" (UniqueName: \"kubernetes.io/projected/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a-kube-api-access-bfgz9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:55.486525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.486439    5098 generic.go:356] "Generic (PLEG): container finished" podID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerID="e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d" exitCode=0
Oct 31 03:48:55.486525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.486500    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"
Oct 31 03:48:55.486806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.486529    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" event={"ID":"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a","Type":"ContainerDied","Data":"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"}
Oct 31 03:48:55.486806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.486597    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4" event={"ID":"5bd9e3a9-0f94-493f-9d4c-18e6302ae53a","Type":"ContainerDied","Data":"4890e398a0df71edc5cd1551749893c4a5a47ed6be0f9db93f7bafcd84837d98"}
Oct 31 03:48:55.486806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.486640    5098 scope.go:122] "RemoveContainer" containerID="e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"
Oct 31 03:48:55.501007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.500987    5098 scope.go:122] "RemoveContainer" containerID="e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"
Oct 31 03:48:55.501323 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:48:55.501305    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d\": container with ID starting with e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d not found: ID does not exist" containerID="e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"
Oct 31 03:48:55.501375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.501331    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d"} err="failed to get container status \"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d\": rpc error: code = NotFound desc = could not find container \"e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d\": container with ID starting with e09e798569a01d9e4d794cf3223abc9a73c1e1c9fa64ca91de3b90f2cffd267d not found: ID does not exist"
Oct 31 03:48:55.545734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.543586    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:48:55.585627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.580017    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-73e222-openshift-default-774bf996f5-vg2l4"]
Oct 31 03:48:55.641309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:55.641270    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" path="/var/lib/kubelet/pods/5bd9e3a9-0f94-493f-9d4c-18e6302ae53a/volumes"
Oct 31 03:48:58.518054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.517958    5098 generic.go:356] "Generic (PLEG): container finished" podID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerID="b08440013915e9d6100ab2dab5dc8de865394a14fe2ee7b37f4f6536a1416a4e" exitCode=0
Oct 31 03:48:58.518054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.518029    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" event={"ID":"b0789148-2f28-4eab-af5b-eb1376e697ab","Type":"ContainerDied","Data":"b08440013915e9d6100ab2dab5dc8de865394a14fe2ee7b37f4f6536a1416a4e"}
Oct 31 03:48:58.839073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.839043    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:48:58.943498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943462    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.943761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943600    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.943761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943717    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.943761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943745    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.943951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943784    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.943951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943877    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.943951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943925    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.944107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.943977    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.944107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944083    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.944107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944097    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.944281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944114    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.944281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944266    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\") pod \"b0789148-2f28-4eab-af5b-eb1376e697ab\" (UID: \"b0789148-2f28-4eab-af5b-eb1376e697ab\") "
Oct 31 03:48:58.944396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944365    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:48:58.944542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.944513    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.945512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.945485    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.945657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.945515    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.945657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.945531    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.945657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.945546    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.945657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.945562    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0789148-2f28-4eab-af5b-eb1376e697ab-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:58.946356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.946318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:58.946461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.946368    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:48:58.946461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.946435    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "kube-api-access-58f5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:48:58.946665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:58.946593    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo" pod "b0789148-2f28-4eab-af5b-eb1376e697ab" (UID: "b0789148-2f28-4eab-af5b-eb1376e697ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:48:59.047432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.047394    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:59.047432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.047427    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:59.047432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.047439    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58f5d\" (UniqueName: \"kubernetes.io/projected/b0789148-2f28-4eab-af5b-eb1376e697ab-kube-api-access-58f5d\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:59.047753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.047459    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0789148-2f28-4eab-af5b-eb1376e697ab-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:48:59.529088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.529056    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"
Oct 31 03:48:59.529088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.529083    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6" event={"ID":"b0789148-2f28-4eab-af5b-eb1376e697ab","Type":"ContainerDied","Data":"aaa6fbcefce3d21e802cd43fc65d4f9e231e5271caec1b5d0991080e93ec7e14"}
Oct 31 03:48:59.529685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.529154    5098 scope.go:122] "RemoveContainer" containerID="b08440013915e9d6100ab2dab5dc8de865394a14fe2ee7b37f4f6536a1416a4e"
Oct 31 03:48:59.556914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.556881    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:48:59.561739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.561715    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-7qjj6"]
Oct 31 03:48:59.615153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.615116    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" path="/var/lib/kubelet/pods/b0789148-2f28-4eab-af5b-eb1376e697ab/volumes"
Oct 31 03:48:59.838130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.838036    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:48:59.839530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839500    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839742    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839786    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839795    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839857    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" containerName="maas-api"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.839864    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" containerName="maas-api"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.840176    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad699c26-7f19-4bcb-8505-397e23a3ceb5" containerName="maas-api"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.840210    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0789148-2f28-4eab-af5b-eb1376e697ab" containerName="istio-proxy"
Oct 31 03:48:59.842055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.840222    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bd9e3a9-0f94-493f-9d4c-18e6302ae53a" containerName="istio-proxy"
Oct 31 03:48:59.847009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.846978    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.851561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.851530    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:48:59.856410 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.856379    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d4145b\"/\"openshift-service-ca.crt\""
Oct 31 03:48:59.856539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.856435    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-d4145b\"/\"kube-root-ca.crt\""
Oct 31 03:48:59.856539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.856476    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d4145b\"/\"default-dockercfg-hksmw\""
Oct 31 03:48:59.857158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.857140    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-d4145b\"/\"auth-test-model-d4145b-kserve-self-signed-certs\""
Oct 31 03:48:59.977129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977093    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzjnf\" (UniqueName: \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.977370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977342    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.977470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977448    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.977655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977601    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.977784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977769    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:48:59.977947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:48:59.977930    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.095700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.088052    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.096076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.096047    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzjnf\" (UniqueName: \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.096240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.096222    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.096398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.096384    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.096678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.096652    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.103629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.096913    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.104492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.104451    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.114627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.104897    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.114627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.093860    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.126631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.126225    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.139515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.139443    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzjnf\" (UniqueName: \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.140516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.140487    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\") pod \"auth-test-model-d4145b-kserve-6969f59f65-t4xkd\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") " pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.163937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.163583    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:00.588945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:00.588911    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:49:00.589402 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:49:00.589096    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod674ae73c_9935_4daf_8f92_02209cd83fd7.slice/crio-59dbcab7a0cdcc203b1456270e09b45e3315c8c11ba8a79441cb570a0f7527d2 WatchSource:0}: Error finding container 59dbcab7a0cdcc203b1456270e09b45e3315c8c11ba8a79441cb570a0f7527d2: Status 404 returned error can't find the container with id 59dbcab7a0cdcc203b1456270e09b45e3315c8c11ba8a79441cb570a0f7527d2
Oct 31 03:49:01.060661 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:49:01.060619    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:49:01.625043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:01.625001    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" event={"ID":"674ae73c-9935-4daf-8f92-02209cd83fd7","Type":"ContainerStarted","Data":"fa48fdb2e316f4751a30a3287f787844385abe8fddaca1c2a435f5d767d5ee2e"}
Oct 31 03:49:01.625043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:01.625050    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" event={"ID":"674ae73c-9935-4daf-8f92-02209cd83fd7","Type":"ContainerStarted","Data":"59dbcab7a0cdcc203b1456270e09b45e3315c8c11ba8a79441cb570a0f7527d2"}
Oct 31 03:49:01.643242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:01.643187    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" podStartSLOduration=2.643172765 podStartE2EDuration="2.643172765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:01.639112593 +0000 UTC m=+2016.164332183" watchObservedRunningTime="2026-10-31 03:49:01.643172765 +0000 UTC m=+2016.168392387"
Oct 31 03:49:02.165046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:02.165013    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:09.754184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.754087    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:49:09.761663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.761634    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.769661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.766987    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d04aad\"/\"kube-root-ca.crt\""
Oct 31 03:49:09.769661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.767050    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:49:09.769661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.767223    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d04aad\"/\"default-dockercfg-qsvhg\""
Oct 31 03:49:09.769661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.767302    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d04aad\"/\"openshift-service-ca.crt\""
Oct 31 03:49:09.769661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.767383    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d04aad\"/\"auth-test-model-d04aad-kserve-self-signed-certs\""
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777106    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777288    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrcp\" (UniqueName: \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777406    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777437    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777473    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.778059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.777910    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879361    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879465    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrcp\" (UniqueName: \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879540    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879560    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879583    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.879793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879723    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.880051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.879982    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.880097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.880063    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.880154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.880131    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.881985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.881963    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.882159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.882141    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:09.888052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:09.888026    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrcp\" (UniqueName: \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\") pod \"auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") " pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:10.084678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.084152    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:10.164902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.164406    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:10.194425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.192393    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:10.405069 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:49:10.405035    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37f143ee_1fd5_4aa1_ac2e_bcd0b3741114.slice/crio-664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3 WatchSource:0}: Error finding container 664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3: Status 404 returned error can't find the container with id 664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3
Oct 31 03:49:10.406240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.406214    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:49:10.713238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.713140    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" event={"ID":"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114","Type":"ContainerStarted","Data":"d383f31808b9c5a86b45296e33f462687f9d4e144b04ac44f1e42da7345a660b"}
Oct 31 03:49:10.713238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.713196    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" event={"ID":"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114","Type":"ContainerStarted","Data":"664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3"}
Oct 31 03:49:10.724845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.724815    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:49:10.735182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:10.735131    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" podStartSLOduration=1.735115602 podStartE2EDuration="1.735115602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:49:10.731082021 +0000 UTC m=+2025.256301611" watchObservedRunningTime="2026-10-31 03:49:10.735115602 +0000 UTC m=+2025.260335191"
Oct 31 03:49:11.086692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:11.086656    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:20.087354 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:20.087322    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:20.100370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:20.100342    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:20.834832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:20.834800    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:49:31.058515 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:49:31.058466    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:49:49.695361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:49:49.695325    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:50:00.836850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:00.836788    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:50:00.837371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:00.836894    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:50:01.063559 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:50:01.063520    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:50:03.762171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.762127    5098 generic.go:356] "Generic (PLEG): container finished" podID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerID="e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab" exitCode=0
Oct 31 03:50:03.764797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.762205    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerDied","Data":"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"}
Oct 31 03:50:03.764797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.762849    5098 scope.go:122] "RemoveContainer" containerID="e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"
Oct 31 03:50:03.765416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.764944    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:03.765416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.765032    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:03.765416 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:03.765078    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:04.782549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.782510    5098 generic.go:356] "Generic (PLEG): container finished" podID="944b1f85-9632-424d-b388-454fea87b870" containerID="eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0" exitCode=0
Oct 31 03:50:04.783031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.782580    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerDied","Data":"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"}
Oct 31 03:50:04.783117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.783098    5098 scope.go:122] "RemoveContainer" containerID="eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"
Oct 31 03:50:04.785207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.785185    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerStarted","Data":"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"}
Oct 31 03:50:04.785659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.785579    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:04.785777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.785682    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:04.785777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.785736    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:04.897469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.897388    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:50:04.897823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.897794    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:04.897925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:04.897845    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:05.823679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:05.822513    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerStarted","Data":"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"}
Oct 31 03:50:05.900631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:05.900079    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:05.900631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:05.900142    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:06.622756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:06.622721    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:50:06.623510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:06.623486    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:50:06.623570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:06.623551    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:50:06.900196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:06.900099    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:06.900196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:06.900186    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:07.623386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:07.623358    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:50:07.623671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:07.623644    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:50:07.914864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:07.914767    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:07.915510 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:07.915471    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:08.623164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:08.623130    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:50:08.623356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:08.623184    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:50:08.904688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:08.897703    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:08.904688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:08.897854    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:09.624824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:09.624786    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:50:09.637779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:09.637731    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:50:09.662219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:09.662189    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:50:09.898082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:09.897989    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body=
Oct 31 03:50:09.898368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:09.898334    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused"
Oct 31 03:50:10.623807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:10.623774    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body=
Oct 31 03:50:10.624021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:10.623837    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused"
Oct 31 03:50:10.902515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:10.902432    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:50:10.902978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:10.902810    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:50:10.903444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:10.903423    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:50:11.739263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:11.739229    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:50:11.743861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:11.743820    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:50:12.787169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.787137    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:50:12.792640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.792590    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.808006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.807973    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:50:12.894998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.894950    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895017    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895173    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895264    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895389    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hzgr\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895520    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895566    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895644    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:12.895723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:12.895696    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.002765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.002721    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.003508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.003488    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.004065 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.004048    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hzgr\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.005725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.005683    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.005947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.005929    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.006131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.006098    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.006258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.006158    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.006258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.006195    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.006258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.006246    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.006702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.003899    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.007115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.007090    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.007266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.003395    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.007728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.007675    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.008587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.008566    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.028713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.018359    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.028713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.028527    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.033553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.033187    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.035500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.035438    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hzgr\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.111214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.110708    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:13.404315 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:13.404274    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e7c907e_d182_499e_903d_f57ff34e667f.slice/crio-bc6acd32a61a0a4b91c03953923a21a09c40f47830376ef7f87e14a64f9d6d2c WatchSource:0}: Error finding container bc6acd32a61a0a4b91c03953923a21a09c40f47830376ef7f87e14a64f9d6d2c: Status 404 returned error can't find the container with id bc6acd32a61a0a4b91c03953923a21a09c40f47830376ef7f87e14a64f9d6d2c
Oct 31 03:50:13.405574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.405546    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:50:13.407818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.407786    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:13.407926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.407854    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:13.407926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.407895    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:13.992712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.992658    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" event={"ID":"5e7c907e-d182-499e-903d-f57ff34e667f","Type":"ContainerStarted","Data":"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"}
Oct 31 03:50:13.993220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:13.992734    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" event={"ID":"5e7c907e-d182-499e-903d-f57ff34e667f","Type":"ContainerStarted","Data":"bc6acd32a61a0a4b91c03953923a21a09c40f47830376ef7f87e14a64f9d6d2c"}
Oct 31 03:50:14.015163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:14.015108    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podStartSLOduration=2.015093805 podStartE2EDuration="2.015093805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:14.012120679 +0000 UTC m=+2088.537340281" watchObservedRunningTime="2026-10-31 03:50:14.015093805 +0000 UTC m=+2088.540313394"
Oct 31 03:50:14.111537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:14.111502    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:14.113506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:14.113478    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:14.113692 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:14.113527    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:15.111983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:15.111950    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:15.112393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:15.112004    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:16.112006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:16.111974    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:16.112397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:16.112025    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:17.111619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:17.111582    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:17.111837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:17.111701    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:18.120632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:18.115045    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:18.120632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:18.115145    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:19.111366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:19.111332    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:19.111560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:19.111396    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:20.111485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.111447    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused" start-of-body=
Oct 31 03:50:20.111958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.111519    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.246:15021/healthz/ready\": dial tcp 10.132.0.246:15021: connect: connection refused"
Oct 31 03:50:20.799937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.799891    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"]
Oct 31 03:50:20.804991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.804965    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.809977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.809948    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:20.810221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.810201    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:50:20.812089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.812063    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"]
Oct 31 03:50:20.890257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.890218    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.890444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.890269    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8km74\" (UniqueName: \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.890556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.890542    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.992486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.992451    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.992687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.992496    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8km74\" (UniqueName: \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.992786 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.992725    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.993211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.993185    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:20.995252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:20.995226    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:21.000562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.000536    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8km74\" (UniqueName: \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\") pod \"maas-api-revoke-sub-0736dc06-zt6pf\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:21.116628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.116498    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:21.117833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.117646    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:21.118369 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.118347    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:50:21.131350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.131323    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:21.433677 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:21.433601    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1d3d7d_10af_4086_86a0_4222364786f5.slice/crio-a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281 WatchSource:0}: Error finding container a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281: Status 404 returned error can't find the container with id a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281
Oct 31 03:50:21.439596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:21.439564    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"]
Oct 31 03:50:22.067199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.067164    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:50:22.072362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.072342    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.087167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.087143    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-d4145b-openshift-default-dockercfg-ldpr2\""
Oct 31 03:50:22.103399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.103366    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:50:22.104680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.104648    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf" event={"ID":"bc1d3d7d-10af-4086-86a0-4222364786f5","Type":"ContainerStarted","Data":"9475ac2f430449f2451875371243959990e2f55571b53789257d0e6f66ae34c9"}
Oct 31 03:50:22.104839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.104700    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf" event={"ID":"bc1d3d7d-10af-4086-86a0-4222364786f5","Type":"ContainerStarted","Data":"a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281"}
Oct 31 03:50:22.160220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160182    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160231    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j6kv\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160383    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160500    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160641    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160668    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.160828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160815    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.161108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160858    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.161108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.160927    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.184251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.184164    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf" podStartSLOduration=2.184143949 podStartE2EDuration="2.184143949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:22.169849367 +0000 UTC m=+2096.695068957" watchObservedRunningTime="2026-10-31 03:50:22.184143949 +0000 UTC m=+2096.709363541"
Oct 31 03:50:22.262123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262082    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262134    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262170    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262353    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262380    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j6kv\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262435    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262582    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262775    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262583    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262874    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262896    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.262937 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.262901    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.263182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.263157    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.263248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.263193    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.264881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.264859    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.265020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.265002    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.270533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.270503    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.270898 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.270882    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j6kv\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\") pod \"e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") " pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.388465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.388360    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:22.723932 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:22.723897    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf274d_d1ed_4778_b674_cc6ad7e4b82e.slice/crio-d43298cd0ed4357932b6eca7a178f9d22e8bd4bbe2dfd49b4ed6023b2badee8d WatchSource:0}: Error finding container d43298cd0ed4357932b6eca7a178f9d22e8bd4bbe2dfd49b4ed6023b2badee8d: Status 404 returned error can't find the container with id d43298cd0ed4357932b6eca7a178f9d22e8bd4bbe2dfd49b4ed6023b2badee8d
Oct 31 03:50:22.724139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.724114    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:50:22.728131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.728100    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:22.728221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.728161    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:22.728221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:22.728193    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:23.115260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.115226    5098 generic.go:356] "Generic (PLEG): container finished" podID="bc1d3d7d-10af-4086-86a0-4222364786f5" containerID="9475ac2f430449f2451875371243959990e2f55571b53789257d0e6f66ae34c9" exitCode=0
Oct 31 03:50:23.115454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.115323    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf" event={"ID":"bc1d3d7d-10af-4086-86a0-4222364786f5","Type":"ContainerDied","Data":"9475ac2f430449f2451875371243959990e2f55571b53789257d0e6f66ae34c9"}
Oct 31 03:50:23.117242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.117217    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" event={"ID":"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e","Type":"ContainerStarted","Data":"b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41"}
Oct 31 03:50:23.117395 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.117260    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" event={"ID":"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e","Type":"ContainerStarted","Data":"d43298cd0ed4357932b6eca7a178f9d22e8bd4bbe2dfd49b4ed6023b2badee8d"}
Oct 31 03:50:23.161173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.161107    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podStartSLOduration=1.161088206 podStartE2EDuration="1.161088206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:23.153331528 +0000 UTC m=+2097.678551118" watchObservedRunningTime="2026-10-31 03:50:23.161088206 +0000 UTC m=+2097.686307798"
Oct 31 03:50:23.389282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.389252    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:23.391446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.391418    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:23.391585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:23.391467    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:24.373883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.373842    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:24.389629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.389534    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:24.389791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.389596    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:24.505931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.505897    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\") pod \"bc1d3d7d-10af-4086-86a0-4222364786f5\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") "
Oct 31 03:50:24.506125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.506104    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\") pod \"bc1d3d7d-10af-4086-86a0-4222364786f5\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") "
Oct 31 03:50:24.506223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.506188    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\" (UniqueName: \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\") pod \"bc1d3d7d-10af-4086-86a0-4222364786f5\" (UID: \"bc1d3d7d-10af-4086-86a0-4222364786f5\") "
Oct 31 03:50:24.506343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.506310    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca" pod "bc1d3d7d-10af-4086-86a0-4222364786f5" (UID: "bc1d3d7d-10af-4086-86a0-4222364786f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:24.506764 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.506740    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:24.508914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.508871    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca" pod "bc1d3d7d-10af-4086-86a0-4222364786f5" (UID: "bc1d3d7d-10af-4086-86a0-4222364786f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:24.509050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.508969    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74" pod "bc1d3d7d-10af-4086-86a0-4222364786f5" (UID: "bc1d3d7d-10af-4086-86a0-4222364786f5"). InnerVolumeSpecName "kube-api-access-8km74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:24.625487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.625447    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc1d3d7d-10af-4086-86a0-4222364786f5-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:24.632639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:24.625824    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8km74\" (UniqueName: \"kubernetes.io/projected/bc1d3d7d-10af-4086-86a0-4222364786f5-kube-api-access-8km74\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:25.136859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:25.136823    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"
Oct 31 03:50:25.137034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:25.136854    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf" event={"ID":"bc1d3d7d-10af-4086-86a0-4222364786f5","Type":"ContainerDied","Data":"a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281"}
Oct 31 03:50:25.137034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:25.136907    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e0eb4622e38edaf652ddf26f55aea4571ee3f40b81224e99edf62cc5120281"
Oct 31 03:50:25.388955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:25.388867    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:25.388955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:25.388927    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:26.247969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.247929    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:50:26.263661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.249096    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bc1d3d7d-10af-4086-86a0-4222364786f5" containerName="revoke-keys"
Oct 31 03:50:26.263661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.249126    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1d3d7d-10af-4086-86a0-4222364786f5" containerName="revoke-keys"
Oct 31 03:50:26.263661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.249330    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1d3d7d-10af-4086-86a0-4222364786f5" containerName="revoke-keys"
Oct 31 03:50:26.277228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.274712    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.282771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.282737    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b81d65-openshift-default-dockercfg-58kth\""
Oct 31 03:50:26.282953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.282869    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:50:26.383971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.383934    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.383971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.383982    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgtgj\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384313    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384498    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384556    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384690    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384818    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.384919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384898    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.385009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.384987    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.388986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.388961    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:26.389341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.389004    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:26.486525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486484    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.486743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486545    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.486743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486662    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.486897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486791    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.486897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486846    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486942    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486982    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.486993    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487024    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgtgj\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487033    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487264    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487424    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487703    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.487879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.487795    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.489446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.489423    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.489925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.489907    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.496813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.496790    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.497071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.497049    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgtgj\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\") pod \"e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") " pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:26.613075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:26.612975    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:27.043506 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:27.043468    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252d74ea_9921_4058_8ca6_1d27167967fa.slice/crio-ebddd2094153d03e1a43955f9c8f70d4763416e407e5ba14c5d394275305e631 WatchSource:0}: Error finding container ebddd2094153d03e1a43955f9c8f70d4763416e407e5ba14c5d394275305e631: Status 404 returned error can't find the container with id ebddd2094153d03e1a43955f9c8f70d4763416e407e5ba14c5d394275305e631
Oct 31 03:50:27.044261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.044231    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:50:27.046782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.046749    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:27.046870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.046819    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:27.046870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.046846    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:27.185679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.185639    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" event={"ID":"252d74ea-9921-4058-8ca6-1d27167967fa","Type":"ContainerStarted","Data":"ebddd2094153d03e1a43955f9c8f70d4763416e407e5ba14c5d394275305e631"}
Oct 31 03:50:27.390150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.390053    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:27.391126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:27.390718    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:28.216550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.216512    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" event={"ID":"252d74ea-9921-4058-8ca6-1d27167967fa","Type":"ContainerStarted","Data":"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"}
Oct 31 03:50:28.245168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.245116    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podStartSLOduration=2.245101922 podStartE2EDuration="2.245101922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:28.240129793 +0000 UTC m=+2102.765349384" watchObservedRunningTime="2026-10-31 03:50:28.245101922 +0000 UTC m=+2102.770321510"
Oct 31 03:50:28.394077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.394038    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:28.394698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.394662    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:28.613492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.613458    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:28.630791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.630747    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:28.631132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:28.631103    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:29.389719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:29.389684    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 31 03:50:29.389881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:29.389738    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 31 03:50:29.614454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:29.614413    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:29.614862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:29.614475    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:30.624634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:30.621033    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:30.624634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:30.621094    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:31.203030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:31.202991    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:31.231084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:31.231050    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:31.236486 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:31.236457    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:50:31.425941 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:50:31.425902    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:50:31.614134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:31.614091    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:31.614358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:31.614154    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:32.026283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.026253    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"]
Oct 31 03:50:32.037105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.037073    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.037851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.037821    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"]
Oct 31 03:50:32.042019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.041797    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:32.042019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.041946    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:50:32.074633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.067687    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.074633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.067912    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.074633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.068022    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzf2\" (UniqueName: \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.174577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.174530    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.174840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.174732    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.174840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.174826    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzf2\" (UniqueName: \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.175333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.175294    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.177447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.177425    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.183191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.183168    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzf2\" (UniqueName: \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\") pod \"maas-api-revoke-sub-0b35f400-9vgsl\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.375036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.374939    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:32.617248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.615180    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:32.617248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.615411    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:32.687106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.687071    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"]
Oct 31 03:50:32.687920 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:32.687891    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9390b98_efa3_449f_a5e1_61794ca09c06.slice/crio-3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b WatchSource:0}: Error finding container 3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b: Status 404 returned error can't find the container with id 3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b
Oct 31 03:50:32.808992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.808956    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:50:32.829150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:32.829125    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:50:33.366632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:33.355008    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl" event={"ID":"f9390b98-efa3-449f-a5e1-61794ca09c06","Type":"ContainerStarted","Data":"70021509e2951e6abf0d111af4f8bd6e5e70d27f44dc54a6eb048fbfaf791124"}
Oct 31 03:50:33.366632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:33.355077    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl" event={"ID":"f9390b98-efa3-449f-a5e1-61794ca09c06","Type":"ContainerStarted","Data":"3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b"}
Oct 31 03:50:33.383319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:33.382678    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl" podStartSLOduration=2.382663735 podStartE2EDuration="2.382663735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:33.379962444 +0000 UTC m=+2107.905182030" watchObservedRunningTime="2026-10-31 03:50:33.382663735 +0000 UTC m=+2107.907883324"
Oct 31 03:50:33.614666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:33.614575    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:33.614811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:33.614658    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:34.365329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:34.365237    5098 generic.go:356] "Generic (PLEG): container finished" podID="f9390b98-efa3-449f-a5e1-61794ca09c06" containerID="70021509e2951e6abf0d111af4f8bd6e5e70d27f44dc54a6eb048fbfaf791124" exitCode=0
Oct 31 03:50:34.365489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:34.365341    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl" event={"ID":"f9390b98-efa3-449f-a5e1-61794ca09c06","Type":"ContainerDied","Data":"70021509e2951e6abf0d111af4f8bd6e5e70d27f44dc54a6eb048fbfaf791124"}
Oct 31 03:50:34.613875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:34.613836    5098 patch_prober.go:29] interesting pod/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused" start-of-body=
Oct 31 03:50:34.614276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:34.613901    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.249:15021/healthz/ready\": dial tcp 10.132.0.249:15021: connect: connection refused"
Oct 31 03:50:35.621624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.621584    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:35.622168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.622149    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:35.623069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.623047    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:50:35.651829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.651766    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:35.676633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.676227    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\") pod \"f9390b98-efa3-449f-a5e1-61794ca09c06\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") "
Oct 31 03:50:35.676633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.676320    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\") pod \"f9390b98-efa3-449f-a5e1-61794ca09c06\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") "
Oct 31 03:50:35.676633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.676408    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\" (UniqueName: \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\") pod \"f9390b98-efa3-449f-a5e1-61794ca09c06\" (UID: \"f9390b98-efa3-449f-a5e1-61794ca09c06\") "
Oct 31 03:50:35.680422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.680390    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca" pod "f9390b98-efa3-449f-a5e1-61794ca09c06" (UID: "f9390b98-efa3-449f-a5e1-61794ca09c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:35.690032 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.690004    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca" pod "f9390b98-efa3-449f-a5e1-61794ca09c06" (UID: "f9390b98-efa3-449f-a5e1-61794ca09c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:35.695823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.695789    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2" pod "f9390b98-efa3-449f-a5e1-61794ca09c06" (UID: "f9390b98-efa3-449f-a5e1-61794ca09c06"). InnerVolumeSpecName "kube-api-access-8tzf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:35.792053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.792016    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:35.792053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.792054    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9390b98-efa3-449f-a5e1-61794ca09c06-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:35.792322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:35.792076    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzf2\" (UniqueName: \"kubernetes.io/projected/f9390b98-efa3-449f-a5e1-61794ca09c06-kube-api-access-8tzf2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:36.385882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:36.385849    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"
Oct 31 03:50:36.385882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:36.385870    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl" event={"ID":"f9390b98-efa3-449f-a5e1-61794ca09c06","Type":"ContainerDied","Data":"3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b"}
Oct 31 03:50:36.386094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:36.385920    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d51598af35c90b65621b0f9470385d70b420458f70cb8227144f16ba392f58b"
Oct 31 03:50:45.299281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.299239    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"]
Oct 31 03:50:45.300581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.300554    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f9390b98-efa3-449f-a5e1-61794ca09c06" containerName="revoke-keys"
Oct 31 03:50:45.300581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.300580    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9390b98-efa3-449f-a5e1-61794ca09c06" containerName="revoke-keys"
Oct 31 03:50:45.300846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.300827    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9390b98-efa3-449f-a5e1-61794ca09c06" containerName="revoke-keys"
Oct 31 03:50:45.307039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.307017    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.310827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.310801    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:50:45.311466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.311437    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"]
Oct 31 03:50:45.312864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.312842    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:50:45.416077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.416037    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.416267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.416161    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.416267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.416245    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2jh7\" (UniqueName: \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.517473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.517436    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.517720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.517491    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.517720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.517524    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2jh7\" (UniqueName: \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.518220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.518193    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.519941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.519921    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.526010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.525988    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2jh7\" (UniqueName: \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\") pod \"maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.637955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.637879    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:45.923298 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:45.923266    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a67c304_080c_44b3_af78_42d607cf4b34.slice/crio-9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c WatchSource:0}: Error finding container 9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c: Status 404 returned error can't find the container with id 9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c
Oct 31 03:50:45.924064 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:45.924040    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"]
Oct 31 03:50:46.508770 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:46.508733    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7" event={"ID":"1a67c304-080c-44b3-af78-42d607cf4b34","Type":"ContainerStarted","Data":"244c3b15958d57c3aeb75e52998d2669a375bec4d20992e6a5ae512f8a7071a8"}
Oct 31 03:50:46.509506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:46.508784    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7" event={"ID":"1a67c304-080c-44b3-af78-42d607cf4b34","Type":"ContainerStarted","Data":"9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c"}
Oct 31 03:50:46.524474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:46.524398    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7" podStartSLOduration=1.5243828719999999 podStartE2EDuration="1.524382872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:46.523497681 +0000 UTC m=+2121.048717270" watchObservedRunningTime="2026-10-31 03:50:46.524382872 +0000 UTC m=+2121.049602494"
Oct 31 03:50:48.473671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.473615    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:50:48.478642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.478617    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.485029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.484662    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-63258b-openshift-default-dockercfg-dsf2l\""
Oct 31 03:50:48.490495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.490458    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:50:48.573156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573120    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573169    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573290    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66lj9\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573427 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573338    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573445    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573541    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573574    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573689    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.573748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.573735    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675129    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675251    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675281    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675352    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66lj9\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675416    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675524    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675635    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675666    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.675858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675769    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.676254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.675984    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.676254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.676001    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.676254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.676176    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.676254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.676211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.676470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.676301    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.677792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.677769    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.678169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.678153    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.683807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.683781    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.683942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.683926    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66lj9\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\") pod \"e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") " pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:48.802317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:48.802278    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:49.098161 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:49.098128    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138a0e1e_19ac_4d88_81f4_a5334a9fbc81.slice/crio-871b1de531b257f46b9f57dd9a2a91372e9c9ecf3c4b3cdfcfc0c7051d2165ea WatchSource:0}: Error finding container 871b1de531b257f46b9f57dd9a2a91372e9c9ecf3c4b3cdfcfc0c7051d2165ea: Status 404 returned error can't find the container with id 871b1de531b257f46b9f57dd9a2a91372e9c9ecf3c4b3cdfcfc0c7051d2165ea
Oct 31 03:50:49.099016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.098996    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:50:49.103349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.102691    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:49.103349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.102854    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:49.103349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.102942    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:49.541939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.541903    5098 generic.go:356] "Generic (PLEG): container finished" podID="1a67c304-080c-44b3-af78-42d607cf4b34" containerID="244c3b15958d57c3aeb75e52998d2669a375bec4d20992e6a5ae512f8a7071a8" exitCode=0
Oct 31 03:50:49.542442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.541977    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7" event={"ID":"1a67c304-080c-44b3-af78-42d607cf4b34","Type":"ContainerDied","Data":"244c3b15958d57c3aeb75e52998d2669a375bec4d20992e6a5ae512f8a7071a8"}
Oct 31 03:50:49.544157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.544128    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" event={"ID":"138a0e1e-19ac-4d88-81f4-a5334a9fbc81","Type":"ContainerStarted","Data":"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"}
Oct 31 03:50:49.544260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.544175    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" event={"ID":"138a0e1e-19ac-4d88-81f4-a5334a9fbc81","Type":"ContainerStarted","Data":"871b1de531b257f46b9f57dd9a2a91372e9c9ecf3c4b3cdfcfc0c7051d2165ea"}
Oct 31 03:50:49.582586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.582511    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podStartSLOduration=1.582489946 podStartE2EDuration="1.582489946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:49.577244783 +0000 UTC m=+2124.102464374" watchObservedRunningTime="2026-10-31 03:50:49.582489946 +0000 UTC m=+2124.107709537"
Oct 31 03:50:49.802651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:49.802548    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:50.804252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.804216    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:50.804796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.804275    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:50.886147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.886076    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:50.919482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.919451    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\") pod \"1a67c304-080c-44b3-af78-42d607cf4b34\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") "
Oct 31 03:50:50.919682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.919500    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\") pod \"1a67c304-080c-44b3-af78-42d607cf4b34\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") "
Oct 31 03:50:50.919774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.919752    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\" (UniqueName: \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\") pod \"1a67c304-080c-44b3-af78-42d607cf4b34\" (UID: \"1a67c304-080c-44b3-af78-42d607cf4b34\") "
Oct 31 03:50:50.919997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.919971    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca" pod "1a67c304-080c-44b3-af78-42d607cf4b34" (UID: "1a67c304-080c-44b3-af78-42d607cf4b34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:50:50.920807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.920778    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:50.922013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.921989    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7" pod "1a67c304-080c-44b3-af78-42d607cf4b34" (UID: "1a67c304-080c-44b3-af78-42d607cf4b34"). InnerVolumeSpecName "kube-api-access-t2jh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:50:50.922287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:50.922271    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca" pod "1a67c304-080c-44b3-af78-42d607cf4b34" (UID: "1a67c304-080c-44b3-af78-42d607cf4b34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:50:51.022441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.022408    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2jh7\" (UniqueName: \"kubernetes.io/projected/1a67c304-080c-44b3-af78-42d607cf4b34-kube-api-access-t2jh7\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:51.022441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.022438    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a67c304-080c-44b3-af78-42d607cf4b34-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:50:51.600931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.600897    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"
Oct 31 03:50:51.600931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.600912    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7" event={"ID":"1a67c304-080c-44b3-af78-42d607cf4b34","Type":"ContainerDied","Data":"9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c"}
Oct 31 03:50:51.601160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.600974    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ea1d51f3ecff1d939236bba5f7a978b95cb33d564c2cecb22f4880f20eb525c"
Oct 31 03:50:51.803284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.803248    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:51.803480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:51.803301    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:52.803631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.800924    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:50:52.803631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.802416    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1a67c304-080c-44b3-af78-42d607cf4b34" containerName="revoke-keys"
Oct 31 03:50:52.803631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.803451    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a67c304-080c-44b3-af78-42d607cf4b34" containerName="revoke-keys"
Oct 31 03:50:52.804263 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.803794    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a67c304-080c-44b3-af78-42d607cf4b34" containerName="revoke-keys"
Oct 31 03:50:52.809118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.808289    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:52.809118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.808353    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:52.818763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.817825    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:50:52.818763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.817946    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.826966 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.826930    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:50:52.844717 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.839814    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:52.874226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.871296    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:50:52.904821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.904780    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5c64\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.904999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.904936    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.905069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905018    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.905125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905075    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.905186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905153    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.905186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905177    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.905311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905291    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.908664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905577    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:52.908664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:52.905707    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007008    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.007268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007127    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.007268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007226    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007262    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007382    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5c64\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007476    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007523    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.007594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007567    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007643    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.007739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007678    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007740    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007763    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.007826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007791    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.007826 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007823    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.008015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007912    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx2w\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.008015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.007985    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.008123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008017    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.008123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008036    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.008123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008115    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.008350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.008350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008197    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.008350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008235    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.008350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.008303    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.010060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.010037    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.010434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.010417    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.017060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.017030    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5c64\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.017217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.017198    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.110549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110450    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110524    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110565    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110628    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110656    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx2w\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110736    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.110825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110782    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110843    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.110922    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.111021    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.111057    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.111106    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.111313    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.111415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.111395    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.113271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.113241    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.113271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.113254    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.119324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.119301    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.119464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.119429    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx2w\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\") pod \"e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") " pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.144498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.144461    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:53.214014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.213982    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:53.481216 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:53.481186    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab131ed9_5f98_4b29_98ea_88a4f42137a8.slice/crio-c1bd528ab365ab15537f8e7aace92017be3156011a87d5a5d632b6d605da8ca1 WatchSource:0}: Error finding container c1bd528ab365ab15537f8e7aace92017be3156011a87d5a5d632b6d605da8ca1: Status 404 returned error can't find the container with id c1bd528ab365ab15537f8e7aace92017be3156011a87d5a5d632b6d605da8ca1
Oct 31 03:50:53.485971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.485746    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.485971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.485835    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.485971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.485879    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.487275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.487249    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:50:53.579172 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:50:53.579138    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4114df17_feac_45ef_b7d7_42d39aaa39fd.slice/crio-cc7a639ae5486f8afd47bc4f3b87bdc0e2387de29086ae1b02b55bf4a7a8f1c7 WatchSource:0}: Error finding container cc7a639ae5486f8afd47bc4f3b87bdc0e2387de29086ae1b02b55bf4a7a8f1c7: Status 404 returned error can't find the container with id cc7a639ae5486f8afd47bc4f3b87bdc0e2387de29086ae1b02b55bf4a7a8f1c7
Oct 31 03:50:53.582656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.582615    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.582783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.582698    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.582783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.582738    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:50:53.583729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.583702    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:50:53.628443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.628407    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" event={"ID":"ab131ed9-5f98-4b29-98ea-88a4f42137a8","Type":"ContainerStarted","Data":"579f511660a1fd83d64101cff99bfcafd3a1266fc6765bcaadafd2e442771351"}
Oct 31 03:50:53.628589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.628472    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" event={"ID":"ab131ed9-5f98-4b29-98ea-88a4f42137a8","Type":"ContainerStarted","Data":"c1bd528ab365ab15537f8e7aace92017be3156011a87d5a5d632b6d605da8ca1"}
Oct 31 03:50:53.630450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.630417    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" event={"ID":"4114df17-feac-45ef-b7d7-42d39aaa39fd","Type":"ContainerStarted","Data":"cc7a639ae5486f8afd47bc4f3b87bdc0e2387de29086ae1b02b55bf4a7a8f1c7"}
Oct 31 03:50:53.678679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.674760    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podStartSLOduration=1.674736359 podStartE2EDuration="1.674736359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:53.648018078 +0000 UTC m=+2128.173237671" watchObservedRunningTime="2026-10-31 03:50:53.674736359 +0000 UTC m=+2128.199955952"
Oct 31 03:50:53.805380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.805351    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:53.805843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:53.805817    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:54.146695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.145785    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:50:54.146885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.146756    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:54.146885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.146810    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:54.642183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.642143    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" event={"ID":"4114df17-feac-45ef-b7d7-42d39aaa39fd","Type":"ContainerStarted","Data":"7352a298d00977dff2344efa58357f229435e807df2e5c90355882ba52253506"}
Oct 31 03:50:54.693632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.692954    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podStartSLOduration=2.692933422 podStartE2EDuration="2.692933422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:50:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:50:54.691136139 +0000 UTC m=+2129.216355789" watchObservedRunningTime="2026-10-31 03:50:54.692933422 +0000 UTC m=+2129.218153012"
Oct 31 03:50:54.809174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.809125    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:54.809816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:54.809780    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:55.144947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.144899    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:55.145141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.144967    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:55.218265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.218223    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:50:55.219393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.219351    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:50:55.219699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.219664    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:50:55.807689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.805151    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused" start-of-body=
Oct 31 03:50:55.807689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:55.805236    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.252:15021/healthz/ready\": dial tcp 10.132.0.252:15021: connect: connection refused"
Oct 31 03:50:56.145106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.145003    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:56.145106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.145084    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:56.215678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.215027    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:50:56.215678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.215089    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:50:56.809741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.809708    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:56.810432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.810411    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:56.810995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:56.810974    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:50:57.145763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:57.145675    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:57.145763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:57.145736    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:57.214423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:57.214385    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:50:57.214624 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:57.214437    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:50:58.145454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:58.145411    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:58.145704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:58.145486    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:58.214883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:58.214844    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:50:58.215333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:58.214922    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:50:59.145169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:59.145127    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:50:59.145364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:59.145188    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:50:59.214669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:59.214632    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:50:59.214867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:50:59.214689    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:51:00.145884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:00.145835    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 31 03:51:00.146434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:00.145924    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 31 03:51:00.214952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:00.214896    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 31 03:51:00.215158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:00.214966    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 31 03:51:01.059850 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:51:01.059807    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:51:01.150300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.150264    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:51:01.150835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.150819    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:51:01.151554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.151530    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:51:01.231289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.223540    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:51:01.231289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.225266    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:51:01.231289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:01.226301    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:51:14.027028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:14.026994    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"]
Oct 31 03:51:14.032295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:14.032262    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8979a606-lsv7l"]
Oct 31 03:51:15.615205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:15.615170    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e558e83f-259e-4c74-997c-efe23a24039b" path="/var/lib/kubelet/pods/e558e83f-259e-4c74-997c-efe23a24039b/volumes"
Oct 31 03:51:31.058652 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:51:31.058595    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:51:56.950057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:56.949977    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:51:56.950877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:51:56.950395    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy" containerID="cri-o://55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7" gracePeriod=30
Oct 31 03:52:01.058746 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:52:01.058703    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:52:02.274492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.274462    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:52:02.388997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.388965    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389062    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389100    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389172    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389209    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389383    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.389665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389394    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:02.389773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389490    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:02.389773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389677    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:02.389773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389499    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:02.389773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.389725    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:02.390122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.390095    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.390219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.390201    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\") pod \"14632dd8-04bf-453b-9f1b-2338355eeae6\" (UID: \"14632dd8-04bf-453b-9f1b-2338355eeae6\") "
Oct 31 03:52:02.391010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.390989    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.391010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391009    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.391177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391019    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.391177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391028    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.391177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391036    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14632dd8-04bf-453b-9f1b-2338355eeae6-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.391733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391707    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "kube-api-access-6rqsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:02.391844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.391746    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:02.392202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.392182    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:02.392438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.392418    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy" pod "14632dd8-04bf-453b-9f1b-2338355eeae6" (UID: "14632dd8-04bf-453b-9f1b-2338355eeae6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:02.492483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.492400    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.492483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.492432    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.492483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.492442    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rqsx\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-kube-api-access-6rqsx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.492483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.492453    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14632dd8-04bf-453b-9f1b-2338355eeae6-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:02.536753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.536718    5098 generic.go:356] "Generic (PLEG): container finished" podID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerID="55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7" exitCode=0
Oct 31 03:52:02.536952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.536772    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"
Oct 31 03:52:02.536952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.536803    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" event={"ID":"14632dd8-04bf-453b-9f1b-2338355eeae6","Type":"ContainerDied","Data":"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"}
Oct 31 03:52:02.536952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.536883    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc" event={"ID":"14632dd8-04bf-453b-9f1b-2338355eeae6","Type":"ContainerDied","Data":"c77dc71402b23de7f9e4e17fcb45ce02f2cd5d57eb81515e987a386055af73ac"}
Oct 31 03:52:02.536952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.536906    5098 scope.go:122] "RemoveContainer" containerID="55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"
Oct 31 03:52:02.551040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.551015    5098 scope.go:122] "RemoveContainer" containerID="55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"
Oct 31 03:52:02.551293 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:52:02.551272    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7\": container with ID starting with 55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7 not found: ID does not exist" containerID="55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"
Oct 31 03:52:02.551352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.551300    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7"} err="failed to get container status \"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7\": rpc error: code = NotFound desc = could not find container \"55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7\": container with ID starting with 55c8dc7ba66739582f301de353bb40294d5d659c3785ecb64d950198059a22b7 not found: ID does not exist"
Oct 31 03:52:02.565453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.565422    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:52:02.570648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:02.570625    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-xwfnc"]
Oct 31 03:52:03.615301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:03.615269    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" path="/var/lib/kubelet/pods/14632dd8-04bf-453b-9f1b-2338355eeae6/volumes"
Oct 31 03:52:06.512372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:06.512292    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:52:06.512813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:06.512524    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy" containerID="cri-o://0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053" gracePeriod=30
Oct 31 03:52:07.030839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:07.030801    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"]
Oct 31 03:52:07.036976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:07.036942    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8cc9dfe5-n4jf9"]
Oct 31 03:52:07.614709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:07.614678    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2019f127-4844-447e-99af-c0e788bff310" path="/var/lib/kubelet/pods/2019f127-4844-447e-99af-c0e788bff310/volumes"
Oct 31 03:52:11.834018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.833988    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:52:11.949387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949354    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949416    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949438    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949464    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949560    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949670    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949757    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949822    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.949847 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949820    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:11.950087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.949876    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:11.950087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.950077    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:52:11.950225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.950200    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\") pod \"1c3a4316-41ab-453c-ac70-842d449fc475\" (UID: \"1c3a4316-41ab-453c-ac70-842d449fc475\") "
Oct 31 03:52:11.950332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.950206    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:11.950407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.950228    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:11.951232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.951206    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c3a4316-41ab-453c-ac70-842d449fc475-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:11.951364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.951236    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:11.951364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.951251    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:11.951364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.951266    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:11.951364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.951279    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:11.952160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.952137    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:11.952358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.952336    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:52:11.952463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.952403    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:52:11.952463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:11.952400    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm" pod "1c3a4316-41ab-453c-ac70-842d449fc475" (UID: "1c3a4316-41ab-453c-ac70-842d449fc475"). InnerVolumeSpecName "kube-api-access-szppm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:52:12.053406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.053325    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c3a4316-41ab-453c-ac70-842d449fc475-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:12.053406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.053355    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:12.053406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.053364    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c3a4316-41ab-453c-ac70-842d449fc475-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:12.053406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.053373    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szppm\" (UniqueName: \"kubernetes.io/projected/1c3a4316-41ab-453c-ac70-842d449fc475-kube-api-access-szppm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:52:12.636184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.636151    5098 generic.go:356] "Generic (PLEG): container finished" podID="1c3a4316-41ab-453c-ac70-842d449fc475" containerID="0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053" exitCode=0
Oct 31 03:52:12.636372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.636201    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"
Oct 31 03:52:12.636372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.636238    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerDied","Data":"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"}
Oct 31 03:52:12.636372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.636289    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q" event={"ID":"1c3a4316-41ab-453c-ac70-842d449fc475","Type":"ContainerDied","Data":"a840234bf645f29e7691ecc1feb0f1d477a0292613001798cdd6dada3f382604"}
Oct 31 03:52:12.636372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.636363    5098 scope.go:122] "RemoveContainer" containerID="0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"
Oct 31 03:52:12.649435 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.649410    5098 scope.go:122] "RemoveContainer" containerID="68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"
Oct 31 03:52:12.662818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.662791    5098 scope.go:122] "RemoveContainer" containerID="0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"
Oct 31 03:52:12.663190 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:52:12.663156    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053\": container with ID starting with 0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053 not found: ID does not exist" containerID="0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"
Oct 31 03:52:12.663321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.663200    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053"} err="failed to get container status \"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053\": rpc error: code = NotFound desc = could not find container \"0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053\": container with ID starting with 0660156a80706a7bb28bfd03c7ee6af8a220032b024a70db2606b7e3e391a053 not found: ID does not exist"
Oct 31 03:52:12.663321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.663226    5098 scope.go:122] "RemoveContainer" containerID="68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"
Oct 31 03:52:12.664020 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:52:12.663578    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361\": container with ID starting with 68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361 not found: ID does not exist" containerID="68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"
Oct 31 03:52:12.664020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.663650    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361"} err="failed to get container status \"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361\": rpc error: code = NotFound desc = could not find container \"68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361\": container with ID starting with 68df0ed917b03a566a396b2839bed2ad10a4b18dc05eb2aaa2c753eab5754361 not found: ID does not exist"
Oct 31 03:52:12.666175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.666154    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:52:12.671907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:12.671885    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-hr75q"]
Oct 31 03:52:13.615040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:13.615009    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" path="/var/lib/kubelet/pods/1c3a4316-41ab-453c-ac70-842d449fc475/volumes"
Oct 31 03:52:18.025314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:18.025273    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"]
Oct 31 03:52:18.030271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:18.030243    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8a3b08b-4vspg"]
Oct 31 03:52:19.614655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:19.614621    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19b11db7-1214-4449-8bb2-09d402b62d89" path="/var/lib/kubelet/pods/19b11db7-1214-4449-8bb2-09d402b62d89/volumes"
Oct 31 03:52:31.059083 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:52:31.059046    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:52:39.237207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:39.237170    5098 scope.go:122] "RemoveContainer" containerID="0ad61034c8413af42ee8e9fa41515388af9ea6da6e49ad1f14709463146c658c"
Oct 31 03:52:39.248597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:39.248577    5098 scope.go:122] "RemoveContainer" containerID="277daf68db40c59a6b980a83dadbcc7ac23b139561e4d23c2b76592fa10eda02"
Oct 31 03:52:40.477839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:40.477795    5098 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6csjw_6d23fdfb-3765-488e-bad2-d80a7fe3a162/konnectivity-agent/0.log" limit=1024
Oct 31 03:52:48.722104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:52:48.722074    5098 ???:1] "http: TLS handshake error from 10.0.142.76:37096: EOF"
Oct 31 03:53:01.058905 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:53:01.058863    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:53:31.058431 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:53:31.058395    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:53:34.698761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:34.698715    5098 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6csjw_6d23fdfb-3765-488e-bad2-d80a7fe3a162/konnectivity-agent/0.log" limit=1024
Oct 31 03:53:39.304827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:39.304792    5098 scope.go:122] "RemoveContainer" containerID="d92289d0546ec6fc79d6efb59c24a228b467976dc549cc89b661460853077207"
Oct 31 03:53:39.316738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:39.316717    5098 scope.go:122] "RemoveContainer" containerID="a243195c02f8755b1347e8447dba82d098a15b636b3873a3f6ca99932560f326"
Oct 31 03:53:39.327373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:39.327352    5098 scope.go:122] "RemoveContainer" containerID="306cd1ecfcc29e198063e33bbf4d46856f1cbbed73c20210f5e38ba1d08468bb"
Oct 31 03:53:39.337997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:39.337973    5098 scope.go:122] "RemoveContainer" containerID="52452547b86d78337c2f8e52a9f8f3cce8061daa3c68e93efae92596bd57f43a"
Oct 31 03:53:50.960268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:50.960229    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:53:50.974766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:50.974738    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:53:55.961189 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:55.961145    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" podUID="49cb135a-57ad-4060-8719-3341d1a152d7" containerName="payload-processing" containerID="cri-o://44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91" gracePeriod=25
Oct 31 03:53:55.975285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:55.975248    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing" containerID="cri-o://192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e" gracePeriod=25
Oct 31 03:53:56.278037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.278008    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:53:56.430079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.430048    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\" (UniqueName: \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\") pod \"49cb135a-57ad-4060-8719-3341d1a152d7\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") "
Oct 31 03:53:56.430269 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.430257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\") pod \"49cb135a-57ad-4060-8719-3341d1a152d7\" (UID: \"49cb135a-57ad-4060-8719-3341d1a152d7\") "
Oct 31 03:53:56.430662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.430628    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume" pod "49cb135a-57ad-4060-8719-3341d1a152d7" (UID: "49cb135a-57ad-4060-8719-3341d1a152d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:53:56.430896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.430878    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49cb135a-57ad-4060-8719-3341d1a152d7-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:56.432311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.432290    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x" pod "49cb135a-57ad-4060-8719-3341d1a152d7" (UID: "49cb135a-57ad-4060-8719-3341d1a152d7"). InnerVolumeSpecName "kube-api-access-5ch8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:53:56.532691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.532654    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ch8x\" (UniqueName: \"kubernetes.io/projected/49cb135a-57ad-4060-8719-3341d1a152d7-kube-api-access-5ch8x\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:53:56.653989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.653957    5098 generic.go:356] "Generic (PLEG): container finished" podID="49cb135a-57ad-4060-8719-3341d1a152d7" containerID="44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91" exitCode=0
Oct 31 03:53:56.654182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.654008    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"
Oct 31 03:53:56.654182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.654018    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" event={"ID":"49cb135a-57ad-4060-8719-3341d1a152d7","Type":"ContainerDied","Data":"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"}
Oct 31 03:53:56.654182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.654057    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv" event={"ID":"49cb135a-57ad-4060-8719-3341d1a152d7","Type":"ContainerDied","Data":"1d5c3a2a8eff3225cd3d5f6a4407772ff44eb588b08c3b09c416e034c66a0a2f"}
Oct 31 03:53:56.654182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.654078    5098 scope.go:122] "RemoveContainer" containerID="44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"
Oct 31 03:53:56.668697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.668676    5098 scope.go:122] "RemoveContainer" containerID="44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"
Oct 31 03:53:56.669023 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:53:56.669004    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91\": container with ID starting with 44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91 not found: ID does not exist" containerID="44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"
Oct 31 03:53:56.669076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.669036    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91"} err="failed to get container status \"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91\": rpc error: code = NotFound desc = could not find container \"44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91\": container with ID starting with 44351f6cf036b37a1aa87eb6186a07a5604162aab8c58e8954bbf0a2e6dabf91 not found: ID does not exist"
Oct 31 03:53:56.683187 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.683160    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:53:56.687918 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:56.687898    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-034fe8-5665ddc6cc-srhgv"]
Oct 31 03:53:57.615563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:53:57.615525    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49cb135a-57ad-4060-8719-3341d1a152d7" path="/var/lib/kubelet/pods/49cb135a-57ad-4060-8719-3341d1a152d7/volumes"
Oct 31 03:54:00.060213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:00.060178    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:54:01.059216 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:54:01.059177    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:54:01.665268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:01.665223    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:54:03.786662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.786623    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"]
Oct 31 03:54:03.787566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787549    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.787651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787568    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.787651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787584    5098 cpu_manager.go:414] "containerMap: removing container" podUID="49cb135a-57ad-4060-8719-3341d1a152d7" containerName="payload-processing"
Oct 31 03:54:03.787651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787589    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49cb135a-57ad-4060-8719-3341d1a152d7" containerName="payload-processing"
Oct 31 03:54:03.787651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787617    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.787651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787622    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.787835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787671    5098 cpu_manager.go:414] "containerMap: removing container" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy"
Oct 31 03:54:03.787835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787676    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy"
Oct 31 03:54:03.787900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787857    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="14632dd8-04bf-453b-9f1b-2338355eeae6" containerName="istio-proxy"
Oct 31 03:54:03.787900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787868    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.787900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787877    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="49cb135a-57ad-4060-8719-3341d1a152d7" containerName="payload-processing"
Oct 31 03:54:03.787900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.787885    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3a4316-41ab-453c-ac70-842d449fc475" containerName="istio-proxy"
Oct 31 03:54:03.793773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.793727    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.797848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.797806    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"]
Oct 31 03:54:03.799016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.798984    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:03.799162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.799055    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:54:03.864073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.864023    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmnhr\" (UniqueName: \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.864254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.864220    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.864405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.864362    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.966893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.966854    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmnhr\" (UniqueName: \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.967101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.966977    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.967167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.967100    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.967829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.967804    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.969493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.969467    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:03.977242 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:03.977212    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmnhr\" (UniqueName: \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\") pod \"maas-api-revoke-sub-4f603a02-h6s8d\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:04.117203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.117119    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:04.408045 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:04.408012    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75979d73_8cc2_486e_9fec_de3e54c076df.slice/crio-67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5 WatchSource:0}: Error finding container 67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5: Status 404 returned error can't find the container with id 67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5
Oct 31 03:54:04.408906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.408876    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"]
Oct 31 03:54:04.735818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.735686    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d" event={"ID":"75979d73-8cc2-486e-9fec-de3e54c076df","Type":"ContainerStarted","Data":"67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5"}
Oct 31 03:54:04.878166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.878126    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"]
Oct 31 03:54:04.883728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.883692    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:04.890380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:04.890344    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"]
Oct 31 03:54:05.020361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.020246    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.020559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.020452    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.020668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.020652    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbm2\" (UniqueName: \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.122616 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.122577    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbm2\" (UniqueName: \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.122839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.122752    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.122945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.122927    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.123459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.123432    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.125345 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.125321    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.131559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.131534    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbm2\" (UniqueName: \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\") pod \"maas-api-revoke-sub-b0268511-qcftz\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.205827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.205791    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:05.472213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.472177    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"]
Oct 31 03:54:05.480445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.480414    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.485413 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.485373    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"]
Oct 31 03:54:05.506436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.506402    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"]
Oct 31 03:54:05.509439 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:05.509390    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04c4f7c_074a_4f49_a4f3_59d7a8063f65.slice/crio-9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969 WatchSource:0}: Error finding container 9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969: Status 404 returned error can't find the container with id 9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969
Oct 31 03:54:05.631417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.631382    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.631630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.631513    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khwk\" (UniqueName: \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.631630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.631596    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.733706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.733662    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.733911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.733749    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khwk\" (UniqueName: \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.733911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.733801    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.734541 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.734515    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.746984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.746938    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz" event={"ID":"c04c4f7c-074a-4f49-a4f3-59d7a8063f65","Type":"ContainerStarted","Data":"9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969"}
Oct 31 03:54:05.748699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.748673    5098 generic.go:356] "Generic (PLEG): container finished" podID="75979d73-8cc2-486e-9fec-de3e54c076df" containerID="0727724ccaa942a5f9fb14f6b488bc46398dda413b932340ec06f859cf5b64a0" exitCode=0
Oct 31 03:54:05.748845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.748757    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d" event={"ID":"75979d73-8cc2-486e-9fec-de3e54c076df","Type":"ContainerDied","Data":"0727724ccaa942a5f9fb14f6b488bc46398dda413b932340ec06f859cf5b64a0"}
Oct 31 03:54:05.749785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.749762    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.750010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.749993    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khwk\" (UniqueName: \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\") pod \"maas-api-revoke-sub-d4226c74-6lvvj\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:05.827027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:05.826943    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:06.237791 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:06.237750    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe547e8d_37d6_41cb_b99f_10b0c9218292.slice/crio-956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e WatchSource:0}: Error finding container 956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e: Status 404 returned error can't find the container with id 956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e
Oct 31 03:54:06.240450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:06.240420    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"]
Oct 31 03:54:06.769415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:06.769379    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj" event={"ID":"fe547e8d-37d6-41cb-b99f-10b0c9218292","Type":"ContainerStarted","Data":"5f7c88f93e86426ca1052ed683e8f0edaafaedf7d02de3f02f98c033394dea98"}
Oct 31 03:54:06.769669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:06.769443    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj" event={"ID":"fe547e8d-37d6-41cb-b99f-10b0c9218292","Type":"ContainerStarted","Data":"956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e"}
Oct 31 03:54:06.772668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:06.772633    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz" event={"ID":"c04c4f7c-074a-4f49-a4f3-59d7a8063f65","Type":"ContainerStarted","Data":"03153e251af1c1f809f39ef8a113f3066958959fdd26e4ccf0dafc5db519faa5"}
Oct 31 03:54:06.796151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:06.796090    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj" podStartSLOduration=1.796072456 podStartE2EDuration="1.796072456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:54:06.790310349 +0000 UTC m=+2321.315529938" watchObservedRunningTime="2026-10-31 03:54:06.796072456 +0000 UTC m=+2321.321292045"
Oct 31 03:54:07.143184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.143154    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:07.193016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.192969    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\") pod \"75979d73-8cc2-486e-9fec-de3e54c076df\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") "
Oct 31 03:54:07.193216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.193092    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\") pod \"75979d73-8cc2-486e-9fec-de3e54c076df\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") "
Oct 31 03:54:07.193216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.193183    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\" (UniqueName: \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\") pod \"75979d73-8cc2-486e-9fec-de3e54c076df\" (UID: \"75979d73-8cc2-486e-9fec-de3e54c076df\") "
Oct 31 03:54:07.193489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.193457    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca" pod "75979d73-8cc2-486e-9fec-de3e54c076df" (UID: "75979d73-8cc2-486e-9fec-de3e54c076df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:07.194166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.194134    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:07.204665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.204225    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca" pod "75979d73-8cc2-486e-9fec-de3e54c076df" (UID: "75979d73-8cc2-486e-9fec-de3e54c076df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:07.204665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.204265    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr" pod "75979d73-8cc2-486e-9fec-de3e54c076df" (UID: "75979d73-8cc2-486e-9fec-de3e54c076df"). InnerVolumeSpecName "kube-api-access-bmnhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:07.296663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.296620    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75979d73-8cc2-486e-9fec-de3e54c076df-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:07.296663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.296657    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmnhr\" (UniqueName: \"kubernetes.io/projected/75979d73-8cc2-486e-9fec-de3e54c076df-kube-api-access-bmnhr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:07.785546 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.785507    5098 generic.go:356] "Generic (PLEG): container finished" podID="fe547e8d-37d6-41cb-b99f-10b0c9218292" containerID="5f7c88f93e86426ca1052ed683e8f0edaafaedf7d02de3f02f98c033394dea98" exitCode=0
Oct 31 03:54:07.785785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.785588    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj" event={"ID":"fe547e8d-37d6-41cb-b99f-10b0c9218292","Type":"ContainerDied","Data":"5f7c88f93e86426ca1052ed683e8f0edaafaedf7d02de3f02f98c033394dea98"}
Oct 31 03:54:07.787691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.787636    5098 generic.go:356] "Generic (PLEG): container finished" podID="c04c4f7c-074a-4f49-a4f3-59d7a8063f65" containerID="03153e251af1c1f809f39ef8a113f3066958959fdd26e4ccf0dafc5db519faa5" exitCode=0
Oct 31 03:54:07.787860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.787744    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz" event={"ID":"c04c4f7c-074a-4f49-a4f3-59d7a8063f65","Type":"ContainerDied","Data":"03153e251af1c1f809f39ef8a113f3066958959fdd26e4ccf0dafc5db519faa5"}
Oct 31 03:54:07.789367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.789333    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"
Oct 31 03:54:07.789481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.789365    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d" event={"ID":"75979d73-8cc2-486e-9fec-de3e54c076df","Type":"ContainerDied","Data":"67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5"}
Oct 31 03:54:07.789481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.789419    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67998b314ed0a58e05a280a6ee7cd5711d133db445d72b840b509d7207e67cd5"
Oct 31 03:54:07.983046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:07.983018    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:08.021388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.021344    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\") pod \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") "
Oct 31 03:54:08.021702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.021674    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\") pod \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") "
Oct 31 03:54:08.021844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.021716    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\" (UniqueName: \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\") pod \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\" (UID: \"c04c4f7c-074a-4f49-a4f3-59d7a8063f65\") "
Oct 31 03:54:08.025174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.025142    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca" pod "c04c4f7c-074a-4f49-a4f3-59d7a8063f65" (UID: "c04c4f7c-074a-4f49-a4f3-59d7a8063f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:08.028809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.028775    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca" pod "c04c4f7c-074a-4f49-a4f3-59d7a8063f65" (UID: "c04c4f7c-074a-4f49-a4f3-59d7a8063f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:08.029075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.029031    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2" pod "c04c4f7c-074a-4f49-a4f3-59d7a8063f65" (UID: "c04c4f7c-074a-4f49-a4f3-59d7a8063f65"). InnerVolumeSpecName "kube-api-access-rkbm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:08.053619 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.053568    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:54:08.054288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.053901    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" podUID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" containerName="maas-api" containerID="cri-o://ee1cf95d51466468288d9c5c87a840dbe05243c6312839de67f58d1086ad8907" gracePeriod=30
Oct 31 03:54:08.127063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.124781    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:08.127063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.124819    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbm2\" (UniqueName: \"kubernetes.io/projected/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-kube-api-access-rkbm2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:08.127063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.124836    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c04c4f7c-074a-4f49-a4f3-59d7a8063f65-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:08.872722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.872680    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"]
Oct 31 03:54:08.873742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.873714    5098 generic.go:356] "Generic (PLEG): container finished" podID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" containerID="ee1cf95d51466468288d9c5c87a840dbe05243c6312839de67f58d1086ad8907" exitCode=0
Oct 31 03:54:08.874076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.874056    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" event={"ID":"d8bb65e9-ed5b-4549-870a-55f5f45d77e3","Type":"ContainerDied","Data":"ee1cf95d51466468288d9c5c87a840dbe05243c6312839de67f58d1086ad8907"}
Oct 31 03:54:08.876741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.876715    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz" event={"ID":"c04c4f7c-074a-4f49-a4f3-59d7a8063f65","Type":"ContainerDied","Data":"9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969"}
Oct 31 03:54:08.876866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.876766    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad7ac1d2642fe9c4bbcf69ad012ae7000335bdbaa53289eea86b55e8ecb6969"
Oct 31 03:54:08.876938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.876924    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"
Oct 31 03:54:08.885407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.885382    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-034fe8-s6ld7"]
Oct 31 03:54:08.935994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.935965    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:54:08.936343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.936322    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\") pod \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") "
Oct 31 03:54:08.936697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.936676    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca" pod "d8bb65e9-ed5b-4549-870a-55f5f45d77e3" (UID: "d8bb65e9-ed5b-4549-870a-55f5f45d77e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:08.936984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:08.936971    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.038654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.038012    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\" (UniqueName: \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\") pod \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") "
Oct 31 03:54:09.038654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.038245    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\") pod \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") "
Oct 31 03:54:09.038654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.038301    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") pod \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\" (UID: \"d8bb65e9-ed5b-4549-870a-55f5f45d77e3\") "
Oct 31 03:54:09.067306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.067264    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs" pod "d8bb65e9-ed5b-4549-870a-55f5f45d77e3" (UID: "d8bb65e9-ed5b-4549-870a-55f5f45d77e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:09.068062 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.068030    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls" pod "d8bb65e9-ed5b-4549-870a-55f5f45d77e3" (UID: "d8bb65e9-ed5b-4549-870a-55f5f45d77e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:09.069440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.069415    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp" pod "d8bb65e9-ed5b-4549-870a-55f5f45d77e3" (UID: "d8bb65e9-ed5b-4549-870a-55f5f45d77e3"). InnerVolumeSpecName "kube-api-access-hgbpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:09.140253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.140223    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:09.141103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.141082    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgbpp\" (UniqueName: \"kubernetes.io/projected/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-kube-api-access-hgbpp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.141157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.141108    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.141157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.141120    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bb65e9-ed5b-4549-870a-55f5f45d77e3-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.242110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.242079    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\") pod \"fe547e8d-37d6-41cb-b99f-10b0c9218292\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") "
Oct 31 03:54:09.242342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.242147    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\" (UniqueName: \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\") pod \"fe547e8d-37d6-41cb-b99f-10b0c9218292\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") "
Oct 31 03:54:09.242342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.242329    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\") pod \"fe547e8d-37d6-41cb-b99f-10b0c9218292\" (UID: \"fe547e8d-37d6-41cb-b99f-10b0c9218292\") "
Oct 31 03:54:09.242568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.242539    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca" pod "fe547e8d-37d6-41cb-b99f-10b0c9218292" (UID: "fe547e8d-37d6-41cb-b99f-10b0c9218292"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:09.243444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.243406    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.244704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.244678    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk" pod "fe547e8d-37d6-41cb-b99f-10b0c9218292" (UID: "fe547e8d-37d6-41cb-b99f-10b0c9218292"). InnerVolumeSpecName "kube-api-access-9khwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:09.245240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.245219    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca" pod "fe547e8d-37d6-41cb-b99f-10b0c9218292" (UID: "fe547e8d-37d6-41cb-b99f-10b0c9218292"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:09.373961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.372691    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe547e8d-37d6-41cb-b99f-10b0c9218292-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.373961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.372768    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khwk\" (UniqueName: \"kubernetes.io/projected/fe547e8d-37d6-41cb-b99f-10b0c9218292-kube-api-access-9khwk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:09.615956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.615922    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a67c304-080c-44b3-af78-42d607cf4b34" path="/var/lib/kubelet/pods/1a67c304-080c-44b3-af78-42d607cf4b34/volumes"
Oct 31 03:54:09.887172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.887092    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"
Oct 31 03:54:09.887172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.887134    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj" event={"ID":"d8bb65e9-ed5b-4549-870a-55f5f45d77e3","Type":"ContainerDied","Data":"a11ece6365eee6a1316895547d69f7a5bd4b43c4a780aae49ef8fe28d5790b0a"}
Oct 31 03:54:09.887748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.887201    5098 scope.go:122] "RemoveContainer" containerID="ee1cf95d51466468288d9c5c87a840dbe05243c6312839de67f58d1086ad8907"
Oct 31 03:54:09.888857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.888835    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"
Oct 31 03:54:09.888857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.888848    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj" event={"ID":"fe547e8d-37d6-41cb-b99f-10b0c9218292","Type":"ContainerDied","Data":"956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e"}
Oct 31 03:54:09.888988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.888891    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="956d6a0ff60170caa54d7d76c5468878ac6980a11139645b42d002da83a0e16e"
Oct 31 03:54:09.918736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.918697    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:54:09.928705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:09.928675    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-034fe8-7674cb47c4-q5dtj"]
Oct 31 03:54:11.683627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:11.682181    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" path="/var/lib/kubelet/pods/d8bb65e9-ed5b-4549-870a-55f5f45d77e3/volumes"
Oct 31 03:54:11.691748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:11.691708    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:54:12.838713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:12.838676    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:54:12.840538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:12.840071    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy" containerID="cri-o://cf6caa107bac2bca7daa7597aca9d7d5330ed6b36e6af720f8918f50018d3e81" gracePeriod=30
Oct 31 03:54:12.853319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:12.853290    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:54:12.853584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:12.853535    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy" containerID="cri-o://8ea0c035362c1f2b441ff38f48d52aafb390be049ad4f868fe4f34a9412cc58f" gracePeriod=30
Oct 31 03:54:13.425633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.425057    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"]
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426730    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c04c4f7c-074a-4f49-a4f3-59d7a8063f65" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426754    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04c4f7c-074a-4f49-a4f3-59d7a8063f65" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426769    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fe547e8d-37d6-41cb-b99f-10b0c9218292" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426775    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe547e8d-37d6-41cb-b99f-10b0c9218292" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426823    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" containerName="maas-api"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426828    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" containerName="maas-api"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426863    5098 cpu_manager.go:414] "containerMap: removing container" podUID="75979d73-8cc2-486e-9fec-de3e54c076df" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.426868    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75979d73-8cc2-486e-9fec-de3e54c076df" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.427134    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="75979d73-8cc2-486e-9fec-de3e54c076df" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.427154    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04c4f7c-074a-4f49-a4f3-59d7a8063f65" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.427161    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe547e8d-37d6-41cb-b99f-10b0c9218292" containerName="revoke-keys"
Oct 31 03:54:13.428637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.427175    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bb65e9-ed5b-4549-870a-55f5f45d77e3" containerName="maas-api"
Oct 31 03:54:13.453981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.453555    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"]
Oct 31 03:54:13.453981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.453863    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.500953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.500920    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.501130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.500997    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb626\" (UniqueName: \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.501130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.501073    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.543151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.543120    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:13.543360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.543340    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:54:13.602477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.602438    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.602691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.602519    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb626\" (UniqueName: \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.602691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.602551    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.603223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.603197    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.605234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.605211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.610805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.610784    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb626\" (UniqueName: \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\") pod \"maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:13.776488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:13.776464    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:14.070709 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:14.070674    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb074239_32ed_4da8_a1fe_cce30a4815b4.slice/crio-e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94 WatchSource:0}: Error finding container e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94: Status 404 returned error can't find the container with id e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94
Oct 31 03:54:14.071785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:14.071752    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"]
Oct 31 03:54:14.946238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:14.946198    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84" event={"ID":"cb074239-32ed-4da8-a1fe-cce30a4815b4","Type":"ContainerStarted","Data":"80a0fbb6096eab6abee1c4c0878628d33a216c2a4121e025b63f9a983d74d48a"}
Oct 31 03:54:14.946434 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:14.946261    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84" event={"ID":"cb074239-32ed-4da8-a1fe-cce30a4815b4","Type":"ContainerStarted","Data":"e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94"}
Oct 31 03:54:15.959515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:15.959475    5098 generic.go:356] "Generic (PLEG): container finished" podID="cb074239-32ed-4da8-a1fe-cce30a4815b4" containerID="80a0fbb6096eab6abee1c4c0878628d33a216c2a4121e025b63f9a983d74d48a" exitCode=0
Oct 31 03:54:15.960040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:15.959561    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84" event={"ID":"cb074239-32ed-4da8-a1fe-cce30a4815b4","Type":"ContainerDied","Data":"80a0fbb6096eab6abee1c4c0878628d33a216c2a4121e025b63f9a983d74d48a"}
Oct 31 03:54:16.385461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.385423    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:54:16.495038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.494998    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\" (UniqueName: \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\") pod \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") "
Oct 31 03:54:16.495371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.495343    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\") pod \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\" (UID: \"2a9afa47-8e4b-492e-9088-c12b2dd52cdf\") "
Oct 31 03:54:16.495791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.495757    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume" pod "2a9afa47-8e4b-492e-9088-c12b2dd52cdf" (UID: "2a9afa47-8e4b-492e-9088-c12b2dd52cdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:16.496281 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.496259    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:16.497541 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.497508    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4" pod "2a9afa47-8e4b-492e-9088-c12b2dd52cdf" (UID: "2a9afa47-8e4b-492e-9088-c12b2dd52cdf"). InnerVolumeSpecName "kube-api-access-qwtz4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:16.597889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:16.597852    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtz4\" (UniqueName: \"kubernetes.io/projected/2a9afa47-8e4b-492e-9088-c12b2dd52cdf-kube-api-access-qwtz4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:17.028046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.027836    5098 generic.go:356] "Generic (PLEG): container finished" podID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerID="192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e" exitCode=0
Oct 31 03:54:17.028560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.028204    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"
Oct 31 03:54:17.028560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.028340    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" event={"ID":"2a9afa47-8e4b-492e-9088-c12b2dd52cdf","Type":"ContainerDied","Data":"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"}
Oct 31 03:54:17.028560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.028408    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6" event={"ID":"2a9afa47-8e4b-492e-9088-c12b2dd52cdf","Type":"ContainerDied","Data":"d72ebdecea8f37e9006952833a87ce6ecf18caf536a660071150a0fd5fd5ec1f"}
Oct 31 03:54:17.028560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.028438    5098 scope.go:122] "RemoveContainer" containerID="192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"
Oct 31 03:54:17.047017 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.046980    5098 scope.go:122] "RemoveContainer" containerID="192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"
Oct 31 03:54:17.047391 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:54:17.047371    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e\": container with ID starting with 192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e not found: ID does not exist" containerID="192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"
Oct 31 03:54:17.047477 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.047399    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e"} err="failed to get container status \"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e\": rpc error: code = NotFound desc = could not find container \"192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e\": container with ID starting with 192b61fde56a987258050ce5b317f7bc7403ba5d382e6c4bc75a21b1d30edf6e not found: ID does not exist"
Oct 31 03:54:17.074206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.074158    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:54:17.084472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.084426    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-034fe8-5ddfb4d547-tp9s6"]
Oct 31 03:54:17.241133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.241099    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:17.335193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.335096    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\") pod \"cb074239-32ed-4da8-a1fe-cce30a4815b4\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") "
Oct 31 03:54:17.335193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.335162    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\" (UniqueName: \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\") pod \"cb074239-32ed-4da8-a1fe-cce30a4815b4\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") "
Oct 31 03:54:17.335431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.335282    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\") pod \"cb074239-32ed-4da8-a1fe-cce30a4815b4\" (UID: \"cb074239-32ed-4da8-a1fe-cce30a4815b4\") "
Oct 31 03:54:17.335546 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.335513    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca" pod "cb074239-32ed-4da8-a1fe-cce30a4815b4" (UID: "cb074239-32ed-4da8-a1fe-cce30a4815b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:17.335986 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.335959    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:17.337654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.337580    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca" pod "cb074239-32ed-4da8-a1fe-cce30a4815b4" (UID: "cb074239-32ed-4da8-a1fe-cce30a4815b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:17.337767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.337712    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626" pod "cb074239-32ed-4da8-a1fe-cce30a4815b4" (UID: "cb074239-32ed-4da8-a1fe-cce30a4815b4"). InnerVolumeSpecName "kube-api-access-vb626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:17.438036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.437992    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb626\" (UniqueName: \"kubernetes.io/projected/cb074239-32ed-4da8-a1fe-cce30a4815b4-kube-api-access-vb626\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:17.438036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.438031    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb074239-32ed-4da8-a1fe-cce30a4815b4-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:17.686142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:17.686055    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" path="/var/lib/kubelet/pods/2a9afa47-8e4b-492e-9088-c12b2dd52cdf/volumes"
Oct 31 03:54:18.059227 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.059190    5098 generic.go:356] "Generic (PLEG): container finished" podID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerID="8ea0c035362c1f2b441ff38f48d52aafb390be049ad4f868fe4f34a9412cc58f" exitCode=0
Oct 31 03:54:18.059889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.059272    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" event={"ID":"95b89f6f-0c66-4c51-ae3a-821e1abe3b03","Type":"ContainerDied","Data":"8ea0c035362c1f2b441ff38f48d52aafb390be049ad4f868fe4f34a9412cc58f"}
Oct 31 03:54:18.062911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.062771    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"
Oct 31 03:54:18.063058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.063038    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84" event={"ID":"cb074239-32ed-4da8-a1fe-cce30a4815b4","Type":"ContainerDied","Data":"e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94"}
Oct 31 03:54:18.063976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.063961    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e4a1e5ee386a49c0b817a9a7cba130d8bbfa8f326f6eb09e6ac551b5be8c94"
Oct 31 03:54:18.066921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.066285    5098 generic.go:356] "Generic (PLEG): container finished" podID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerID="cf6caa107bac2bca7daa7597aca9d7d5330ed6b36e6af720f8918f50018d3e81" exitCode=0
Oct 31 03:54:18.066921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.066414    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" event={"ID":"13e29d8b-1433-449a-b73c-c1431fcd4ca5","Type":"ContainerDied","Data":"cf6caa107bac2bca7daa7597aca9d7d5330ed6b36e6af720f8918f50018d3e81"}
Oct 31 03:54:18.143653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.143616    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.281994    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282054    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282141    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282202    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282326    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282542    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282598    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282650    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.286644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.282739    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\") pod \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\" (UID: \"13e29d8b-1433-449a-b73c-c1431fcd4ca5\") "
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.289136    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.289489    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.289854    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.289933    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.290142    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.292425    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "kube-api-access-phs72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.294701    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.295318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:18.307191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.295422    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy" pod "13e29d8b-1433-449a-b73c-c1431fcd4ca5" (UID: "13e29d8b-1433-449a-b73c-c1431fcd4ca5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.365729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.365697    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387299    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387355    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387371    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387384    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387449    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387462    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387494    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387507    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phs72\" (UniqueName: \"kubernetes.io/projected/13e29d8b-1433-449a-b73c-c1431fcd4ca5-kube-api-access-phs72\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.393031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.387522    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13e29d8b-1433-449a-b73c-c1431fcd4ca5-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.488504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488419    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488495    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488538    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488637    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488687    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488717    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.488759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488754    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.489142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488906    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:18.489142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.488955    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.489257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.489013    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.489257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.489046    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.489257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.489128    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.489257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.489206    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\") pod \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\" (UID: \"95b89f6f-0c66-4c51-ae3a-821e1abe3b03\") "
Oct 31 03:54:18.489478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.489380    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.490266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.490237    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.490381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.490267    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.490381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.490283    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.490381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.490298    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.490381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.490311    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.491238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.491213    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:18.491722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.491701    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "kube-api-access-2gwj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:18.491795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.491725    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:18.492757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.492731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy" pod "95b89f6f-0c66-4c51-ae3a-821e1abe3b03" (UID: "95b89f6f-0c66-4c51-ae3a-821e1abe3b03"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:18.592111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.592061    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.592111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.592102    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.592111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.592112    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gwj6\" (UniqueName: \"kubernetes.io/projected/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-kube-api-access-2gwj6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:18.592111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:18.592121    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95b89f6f-0c66-4c51-ae3a-821e1abe3b03-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:19.079706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.079664    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"
Oct 31 03:54:19.079706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.079673    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5" event={"ID":"95b89f6f-0c66-4c51-ae3a-821e1abe3b03","Type":"ContainerDied","Data":"b75ad4b2e0681f080f6f0fa33d8824cc2ecca00807156a04751bbf7ec80fd0c1"}
Oct 31 03:54:19.080300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.079757    5098 scope.go:122] "RemoveContainer" containerID="8ea0c035362c1f2b441ff38f48d52aafb390be049ad4f868fe4f34a9412cc58f"
Oct 31 03:54:19.081795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.081766    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr" event={"ID":"13e29d8b-1433-449a-b73c-c1431fcd4ca5","Type":"ContainerDied","Data":"4d08887c128f274bc25899c48d90af0b83da4c0e367ac2df74b3d522af0307b4"}
Oct 31 03:54:19.081916 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.081841    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"
Oct 31 03:54:19.094489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.094469    5098 scope.go:122] "RemoveContainer" containerID="cf6caa107bac2bca7daa7597aca9d7d5330ed6b36e6af720f8918f50018d3e81"
Oct 31 03:54:19.109662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.109639    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:54:19.115059 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.115033    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-2jjv5"]
Oct 31 03:54:19.124882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.124850    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:54:19.129796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.129772    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-034fe8-openshift-default-76df768955-kcxqr"]
Oct 31 03:54:19.615027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.614993    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" path="/var/lib/kubelet/pods/13e29d8b-1433-449a-b73c-c1431fcd4ca5/volumes"
Oct 31 03:54:19.615573 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:19.615557    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" path="/var/lib/kubelet/pods/95b89f6f-0c66-4c51-ae3a-821e1abe3b03/volumes"
Oct 31 03:54:23.616657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:23.616623    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:54:26.505875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.504267    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"]
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.519993    5098 cpu_manager.go:414] "containerMap: removing container" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520027    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520054    5098 cpu_manager.go:414] "containerMap: removing container" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520062    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520136    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cb074239-32ed-4da8-a1fe-cce30a4815b4" containerName="revoke-keys"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520145    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb074239-32ed-4da8-a1fe-cce30a4815b4" containerName="revoke-keys"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520199    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520207    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520494    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b89f6f-0c66-4c51-ae3a-821e1abe3b03" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520509    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e29d8b-1433-449a-b73c-c1431fcd4ca5" containerName="istio-proxy"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520523    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb074239-32ed-4da8-a1fe-cce30a4815b4" containerName="revoke-keys"
Oct 31 03:54:26.522632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.520545    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9afa47-8e4b-492e-9088-c12b2dd52cdf" containerName="payload-pre-processing"
Oct 31 03:54:26.550877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.550843    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"]
Oct 31 03:54:26.551352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.551192    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.563943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.563913    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:26.564156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.563949    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:54:26.671400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.671363    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.671843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.671816    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5dlx\" (UniqueName: \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.671993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.671964    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.773371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.773271    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5dlx\" (UniqueName: \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.773371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.773364    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.773618 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.773456    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.774210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.774186    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.776079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.776053    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.782118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.782091    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5dlx\" (UniqueName: \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\") pod \"maas-api-revoke-sub-6b10398d-qjsvz\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:26.886748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:26.886709    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:27.179101 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:27.179066    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd107debb_0e4d_4c39_9d8e_5c8e6cfa9401.slice/crio-957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4 WatchSource:0}: Error finding container 957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4: Status 404 returned error can't find the container with id 957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4
Oct 31 03:54:27.180279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:27.180251    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"]
Oct 31 03:54:27.304539 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:27.304493    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz" event={"ID":"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401","Type":"ContainerStarted","Data":"957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4"}
Oct 31 03:54:28.315312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:28.315277    5098 generic.go:356] "Generic (PLEG): container finished" podID="d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" containerID="140be9d648c49f0ef777ab9e08101b81fc3987737d755f2cda2ba195b4a1cebf" exitCode=0
Oct 31 03:54:28.315822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:28.315364    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz" event={"ID":"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401","Type":"ContainerDied","Data":"140be9d648c49f0ef777ab9e08101b81fc3987737d755f2cda2ba195b4a1cebf"}
Oct 31 03:54:28.617685 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:28.616873    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing" containerID="cri-o://ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a" gracePeriod=25
Oct 31 03:54:29.518236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.518204    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:29.673183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.673083    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\") pod \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") "
Oct 31 03:54:29.673183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.673140    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\") pod \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") "
Oct 31 03:54:29.673428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.673414    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\" (UniqueName: \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\") pod \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\" (UID: \"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401\") "
Oct 31 03:54:29.673499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.673479    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca" pod "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" (UID: "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:29.674083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.674061    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:29.675273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.675249    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca" pod "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" (UID: "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:29.675498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.675476    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx" pod "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" (UID: "d107debb-0e4d-4c39-9d8e-5c8e6cfa9401"). InnerVolumeSpecName "kube-api-access-r5dlx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:29.775781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.775739    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5dlx\" (UniqueName: \"kubernetes.io/projected/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-kube-api-access-r5dlx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:29.775781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:29.775774    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:30.334251 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:30.334215    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz" event={"ID":"d107debb-0e4d-4c39-9d8e-5c8e6cfa9401","Type":"ContainerDied","Data":"957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4"}
Oct 31 03:54:30.334439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:30.334276    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="957cb40e501b50a4bf78fd8270e63dcb602243660cfa163fac7f1c577e153ef4"
Oct 31 03:54:30.334439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:30.334231    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"
Oct 31 03:54:31.060010 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:54:31.059972    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:54:35.390668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:35.380680    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"]
Oct 31 03:54:35.435628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:35.423211    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-46ebfe-cmh84"]
Oct 31 03:54:35.689250 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:35.689167    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb074239-32ed-4da8-a1fe-cce30a4815b4" path="/var/lib/kubelet/pods/cb074239-32ed-4da8-a1fe-cce30a4815b4/volumes"
Oct 31 03:54:35.698077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:35.698025    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:54:39.392931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:39.392897    5098 scope.go:122] "RemoveContainer" containerID="0037e028b55f834701ae3357fb02918de2038e99adbb7272722c20a8b3f6bb66"
Oct 31 03:54:39.408792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:39.408770    5098 scope.go:122] "RemoveContainer" containerID="107b6483d5deb90bafb269762fcc2b04d697e903dafbec0334b43c4ace2178c1"
Oct 31 03:54:42.398271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:42.398213    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:54:42.399999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:42.398583    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy" containerID="cri-o://5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c" gracePeriod=30
Oct 31 03:54:45.699301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:45.699255    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:54:46.572840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.572791    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"]
Oct 31 03:54:46.574599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.574562    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" containerName="revoke-keys"
Oct 31 03:54:46.574834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.574818    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" containerName="revoke-keys"
Oct 31 03:54:46.575205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.575189    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" containerName="revoke-keys"
Oct 31 03:54:46.608437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.608404    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.645321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.643302    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:54:46.645321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.644848    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:54:46.651814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.651778    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"]
Oct 31 03:54:46.916464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.837717    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwc4z\" (UniqueName: \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.916464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.837863    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.916464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.837999    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.941239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.941196    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwc4z\" (UniqueName: \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.941599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.941580    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.941893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.941873    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:46.944378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:46.944128    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:47.157563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:47.157516    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:47.186579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:47.167931    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwc4z\" (UniqueName: \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\") pod \"maas-api-revoke-sub-9d9fae4b-55krz\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:47.327231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:47.325915    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:48.004456 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.004419    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"]
Oct 31 03:54:48.010489 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:54:48.010453    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd960d4bd_772c_41aa_b505_b97b38677413.slice/crio-766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0 WatchSource:0}: Error finding container 766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0: Status 404 returned error can't find the container with id 766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0
Oct 31 03:54:48.149131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.149102    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:54:48.244675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.244641    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz" event={"ID":"d960d4bd-772c-41aa-b505-b97b38677413","Type":"ContainerStarted","Data":"766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0"}
Oct 31 03:54:48.246151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.246122    5098 generic.go:356] "Generic (PLEG): container finished" podID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerID="5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c" exitCode=0
Oct 31 03:54:48.246306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.246168    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"
Oct 31 03:54:48.246306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.246206    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" event={"ID":"da1d86cb-9463-42fd-b6eb-3360dbe651f5","Type":"ContainerDied","Data":"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"}
Oct 31 03:54:48.246306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.246260    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq" event={"ID":"da1d86cb-9463-42fd-b6eb-3360dbe651f5","Type":"ContainerDied","Data":"6d608aa294d75515c3795d1b2285cd3b2b38dee67760577086c8ed8f3c0d5d5f"}
Oct 31 03:54:48.246306 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.246298    5098 scope.go:122] "RemoveContainer" containerID="5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"
Oct 31 03:54:48.254161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254137    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254193    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254279    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254343    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254476    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254597    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:48.254780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254710    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:48.254780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254733    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:48.254921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254855    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.254983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.254929    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.255046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255007    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.255108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255076    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\") pod \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\" (UID: \"da1d86cb-9463-42fd-b6eb-3360dbe651f5\") "
Oct 31 03:54:48.255297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255271    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:48.255370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255356    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:48.255897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255872    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.255897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255893    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.256066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255905    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.256066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255914    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.256066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.255923    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.256751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.256725    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:54:48.257011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.256980    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "kube-api-access-k576z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:48.257225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.257197    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:54:48.257850 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.257826    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token" pod "da1d86cb-9463-42fd-b6eb-3360dbe651f5" (UID: "da1d86cb-9463-42fd-b6eb-3360dbe651f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:48.260777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.260753    5098 scope.go:122] "RemoveContainer" containerID="5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"
Oct 31 03:54:48.261074 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:54:48.261050    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c\": container with ID starting with 5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c not found: ID does not exist" containerID="5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"
Oct 31 03:54:48.261179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.261082    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c"} err="failed to get container status \"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c\": rpc error: code = NotFound desc = could not find container \"5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c\": container with ID starting with 5f72e3ddf08c89e7495d134d8ac2c1bcea63d5b473cb5934029bb2153eb0f19c not found: ID does not exist"
Oct 31 03:54:48.362366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.362258    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.362366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.362307    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k576z\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-kube-api-access-k576z\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.362366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.362327    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.362366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.362341    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1d86cb-9463-42fd-b6eb-3360dbe651f5-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:48.596132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.596091    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:54:48.602311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.602274    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-46ebfe-openshift-default-87ff596f-rcqxq"]
Oct 31 03:54:48.898108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.898080    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:54:48.988673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.982220    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\") pod \"cd96482e-2160-438b-a8e0-7ce09f950cf3\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") "
Oct 31 03:54:48.988673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.982347    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\" (UniqueName: \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\") pod \"cd96482e-2160-438b-a8e0-7ce09f950cf3\" (UID: \"cd96482e-2160-438b-a8e0-7ce09f950cf3\") "
Oct 31 03:54:48.998642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:48.993522    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg" pod "cd96482e-2160-438b-a8e0-7ce09f950cf3" (UID: "cd96482e-2160-438b-a8e0-7ce09f950cf3"). InnerVolumeSpecName "kube-api-access-djvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:49.032155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.008511    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume" pod "cd96482e-2160-438b-a8e0-7ce09f950cf3" (UID: "cd96482e-2160-438b-a8e0-7ce09f950cf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:49.122632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.089458    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd96482e-2160-438b-a8e0-7ce09f950cf3-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:49.122632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.089503    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djvsg\" (UniqueName: \"kubernetes.io/projected/cd96482e-2160-438b-a8e0-7ce09f950cf3-kube-api-access-djvsg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:49.372907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.372859    5098 generic.go:356] "Generic (PLEG): container finished" podID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerID="ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a" exitCode=0
Oct 31 03:54:49.372907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.372910    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"
Oct 31 03:54:49.373139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.373108    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" event={"ID":"cd96482e-2160-438b-a8e0-7ce09f950cf3","Type":"ContainerDied","Data":"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"}
Oct 31 03:54:49.373226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.373170    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw" event={"ID":"cd96482e-2160-438b-a8e0-7ce09f950cf3","Type":"ContainerDied","Data":"b9c74272c066973a43d461aeb10625d5ed289ec41034568a0fbf3530210a0ced"}
Oct 31 03:54:49.373226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.373201    5098 scope.go:122] "RemoveContainer" containerID="ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"
Oct 31 03:54:49.375837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.375806    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz" event={"ID":"d960d4bd-772c-41aa-b505-b97b38677413","Type":"ContainerStarted","Data":"ae37b6a1033649aa8c0219fb5070fa14a138548a08e293f999a7f1ca765b1ce3"}
Oct 31 03:54:49.387031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.387009    5098 scope.go:122] "RemoveContainer" containerID="ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"
Oct 31 03:54:49.387382 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:54:49.387362    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a\": container with ID starting with ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a not found: ID does not exist" containerID="ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"
Oct 31 03:54:49.387441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.387394    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a"} err="failed to get container status \"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a\": rpc error: code = NotFound desc = could not find container \"ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a\": container with ID starting with ddd646c0ba9e31f0d00eced06d0a6c20810def3ca2b3558e83db621da8fe1f0a not found: ID does not exist"
Oct 31 03:54:49.417849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.417814    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:54:49.426259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.426221    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-46ebfe-76d66947b7-jmnzw"]
Oct 31 03:54:49.614212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.614120    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" path="/var/lib/kubelet/pods/cd96482e-2160-438b-a8e0-7ce09f950cf3/volumes"
Oct 31 03:54:49.614917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:49.614890    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" path="/var/lib/kubelet/pods/da1d86cb-9463-42fd-b6eb-3360dbe651f5/volumes"
Oct 31 03:54:50.406209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.406175    5098 generic.go:356] "Generic (PLEG): container finished" podID="d960d4bd-772c-41aa-b505-b97b38677413" containerID="ae37b6a1033649aa8c0219fb5070fa14a138548a08e293f999a7f1ca765b1ce3" exitCode=0
Oct 31 03:54:50.406651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.406261    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz" event={"ID":"d960d4bd-772c-41aa-b505-b97b38677413","Type":"ContainerDied","Data":"ae37b6a1033649aa8c0219fb5070fa14a138548a08e293f999a7f1ca765b1ce3"}
Oct 31 03:54:50.587153 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.587118    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:50.658794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.658699    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\" (UniqueName: \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\") pod \"d960d4bd-772c-41aa-b505-b97b38677413\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") "
Oct 31 03:54:50.658953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.658836    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\") pod \"d960d4bd-772c-41aa-b505-b97b38677413\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") "
Oct 31 03:54:50.659111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.659086    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\") pod \"d960d4bd-772c-41aa-b505-b97b38677413\" (UID: \"d960d4bd-772c-41aa-b505-b97b38677413\") "
Oct 31 03:54:50.659310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.659273    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca" pod "d960d4bd-772c-41aa-b505-b97b38677413" (UID: "d960d4bd-772c-41aa-b505-b97b38677413"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:54:50.660165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.660139    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:50.661518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.661490    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca" pod "d960d4bd-772c-41aa-b505-b97b38677413" (UID: "d960d4bd-772c-41aa-b505-b97b38677413"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:54:50.661807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.661783    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z" pod "d960d4bd-772c-41aa-b505-b97b38677413" (UID: "d960d4bd-772c-41aa-b505-b97b38677413"). InnerVolumeSpecName "kube-api-access-fwc4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:54:50.784631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.782493    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d960d4bd-772c-41aa-b505-b97b38677413-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:50.784631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:50.782546    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwc4z\" (UniqueName: \"kubernetes.io/projected/d960d4bd-772c-41aa-b505-b97b38677413-kube-api-access-fwc4z\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:54:51.416432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:51.416394    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz" event={"ID":"d960d4bd-772c-41aa-b505-b97b38677413","Type":"ContainerDied","Data":"766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0"}
Oct 31 03:54:51.416884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:51.416434    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"
Oct 31 03:54:51.416884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:54:51.416450    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="766e11d4f9290b59c405cd35ac6326ee540f21b0534e55e90d1edd388c7b15a0"
Oct 31 03:55:01.061581 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:01.061537    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:55:07.129452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.129350    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"]
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130514    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d960d4bd-772c-41aa-b505-b97b38677413" containerName="revoke-keys"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130536    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d960d4bd-772c-41aa-b505-b97b38677413" containerName="revoke-keys"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130560    5098 cpu_manager.go:414] "containerMap: removing container" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130567    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130580    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130585    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130780    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d960d4bd-772c-41aa-b505-b97b38677413" containerName="revoke-keys"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130793    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1d86cb-9463-42fd-b6eb-3360dbe651f5" containerName="istio-proxy"
Oct 31 03:55:07.132174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.130800    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd96482e-2160-438b-a8e0-7ce09f950cf3" containerName="payload-pre-processing"
Oct 31 03:55:07.135325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.135296    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.141579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.141548    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:07.141781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.141726    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:55:07.145746 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.145713    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"]
Oct 31 03:55:07.197807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.197765    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.197807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.197814    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.198110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.197971    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb79\" (UniqueName: \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.288735 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.288702    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"]
Oct 31 03:55:07.300878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.299360    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb79\" (UniqueName: \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.300878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.299808    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.300878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.299849    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.300878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.300774    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"]
Oct 31 03:55:07.300878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.300878    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.301307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.300996    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.303055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.303023    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.308682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.308650    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb79\" (UniqueName: \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.401355 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.401248    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.401540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.401410    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.401540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.401459    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp4zb\" (UniqueName: \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.468097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.468055    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:07.503602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.503550    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.503840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.503693    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.503840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.503753    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp4zb\" (UniqueName: \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.504458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.504429    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.506182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.506158    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.522556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.522526    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp4zb\" (UniqueName: \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\") pod \"maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.662574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.658924    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:07.706067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.706033    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"]
Oct 31 03:55:07.713118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.713086    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.735586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.729267    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"]
Oct 31 03:55:07.762551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.762505    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl5zm\" (UniqueName: \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.762932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.762904    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.763951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.763259    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.773556 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:07.773520    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4634040b_5664_4610_90de_a2c62230035c.slice/crio-cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f WatchSource:0}: Error finding container cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f: Status 404 returned error can't find the container with id cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f
Oct 31 03:55:07.790007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.779403    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"]
Oct 31 03:55:07.865950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.865907    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.866276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.866250    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl5zm\" (UniqueName: \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.866399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.866306    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.867255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.867226    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.950102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.950000    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.950294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.950176    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl5zm\" (UniqueName: \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:07.998828 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:07.998796    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04e5b8ab_d33f_4af1_866a_e047b98261da.slice/crio-e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1 WatchSource:0}: Error finding container e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1: Status 404 returned error can't find the container with id e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1
Oct 31 03:55:07.999492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:07.999466    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"]
Oct 31 03:55:08.044773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.044743    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:08.500596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.500558    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"]
Oct 31 03:55:08.532198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.532158    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.553331 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.550665    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"]
Oct 31 03:55:08.561248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.561215    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"]
Oct 31 03:55:08.590726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.590697    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.590987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.590960    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.591098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.591082    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq548\" (UniqueName: \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.692699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.692660    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.692915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.692755    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq548\" (UniqueName: \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.692915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.692902    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.693376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.693352    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.709853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.709811    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs" event={"ID":"4634040b-5664-4610-90de-a2c62230035c","Type":"ContainerStarted","Data":"0d21cf78c7a11713920b811a50a385fcd67f640f260735905d61d2b91bc7c6ed"}
Oct 31 03:55:08.710063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.709879    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs" event={"ID":"4634040b-5664-4610-90de-a2c62230035c","Type":"ContainerStarted","Data":"cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f"}
Oct 31 03:55:08.711290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.711264    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth" event={"ID":"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928","Type":"ContainerStarted","Data":"76719e8c3195d55edf5bd251c8be479ef95590f0f459451c71d94adcb7b414bd"}
Oct 31 03:55:08.713276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.713234    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8" event={"ID":"04e5b8ab-d33f-4af1-866a-e047b98261da","Type":"ContainerStarted","Data":"efaeae4b94e7fa487c720eb5f906349e2c3d7c5573063dcb12b8386186010d51"}
Oct 31 03:55:08.713399 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.713295    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8" event={"ID":"04e5b8ab-d33f-4af1-866a-e047b98261da","Type":"ContainerStarted","Data":"e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1"}
Oct 31 03:55:08.735358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.735304    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs" podStartSLOduration=1.7352895099999999 podStartE2EDuration="1.73528951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:08.730485714 +0000 UTC m=+2383.255705305" watchObservedRunningTime="2026-10-31 03:55:08.73528951 +0000 UTC m=+2383.260509099"
Oct 31 03:55:08.750554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.750517    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq548\" (UniqueName: \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.751219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.751193    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:08.752097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.752042    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8" podStartSLOduration=1.752024424 podStartE2EDuration="1.752024424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:08.748912775 +0000 UTC m=+2383.274132365" watchObservedRunningTime="2026-10-31 03:55:08.752024424 +0000 UTC m=+2383.277244013"
Oct 31 03:55:08.884411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:08.884382    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:09.171699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.171665    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"]
Oct 31 03:55:09.201688 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:09.201638    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod886ab726_69c6_41e0_9fd4_237790b9fee3.slice/crio-74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c WatchSource:0}: Error finding container 74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c: Status 404 returned error can't find the container with id 74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c
Oct 31 03:55:09.725080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.725043    5098 generic.go:356] "Generic (PLEG): container finished" podID="4634040b-5664-4610-90de-a2c62230035c" containerID="0d21cf78c7a11713920b811a50a385fcd67f640f260735905d61d2b91bc7c6ed" exitCode=0
Oct 31 03:55:09.725550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.725122    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs" event={"ID":"4634040b-5664-4610-90de-a2c62230035c","Type":"ContainerDied","Data":"0d21cf78c7a11713920b811a50a385fcd67f640f260735905d61d2b91bc7c6ed"}
Oct 31 03:55:09.727825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.727794    5098 generic.go:356] "Generic (PLEG): container finished" podID="4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" containerID="daed7ed9edc111649be6866da33a5c7ced43e0449b8000cc1fd3199fb09f8294" exitCode=0
Oct 31 03:55:09.727983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.727867    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth" event={"ID":"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928","Type":"ContainerDied","Data":"daed7ed9edc111649be6866da33a5c7ced43e0449b8000cc1fd3199fb09f8294"}
Oct 31 03:55:09.729832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.729804    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d" event={"ID":"886ab726-69c6-41e0-9fd4-237790b9fee3","Type":"ContainerStarted","Data":"ac90da13deaf7eab502759e2291bc6f258d3484e7d57a533e0cf058d2c769a27"}
Oct 31 03:55:09.729997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.729857    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d" event={"ID":"886ab726-69c6-41e0-9fd4-237790b9fee3","Type":"ContainerStarted","Data":"74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c"}
Oct 31 03:55:09.731496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.731474    5098 generic.go:356] "Generic (PLEG): container finished" podID="04e5b8ab-d33f-4af1-866a-e047b98261da" containerID="efaeae4b94e7fa487c720eb5f906349e2c3d7c5573063dcb12b8386186010d51" exitCode=0
Oct 31 03:55:09.731630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.731551    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8" event={"ID":"04e5b8ab-d33f-4af1-866a-e047b98261da","Type":"ContainerDied","Data":"efaeae4b94e7fa487c720eb5f906349e2c3d7c5573063dcb12b8386186010d51"}
Oct 31 03:55:09.806066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:09.805998    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d" podStartSLOduration=1.8059797560000002 podStartE2EDuration="1.805979756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:09.80491743 +0000 UTC m=+2384.330137019" watchObservedRunningTime="2026-10-31 03:55:09.805979756 +0000 UTC m=+2384.331199344"
Oct 31 03:55:10.743615 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:10.743566    5098 generic.go:356] "Generic (PLEG): container finished" podID="886ab726-69c6-41e0-9fd4-237790b9fee3" containerID="ac90da13deaf7eab502759e2291bc6f258d3484e7d57a533e0cf058d2c769a27" exitCode=0
Oct 31 03:55:10.744051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:10.743707    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d" event={"ID":"886ab726-69c6-41e0-9fd4-237790b9fee3","Type":"ContainerDied","Data":"ac90da13deaf7eab502759e2291bc6f258d3484e7d57a533e0cf058d2c769a27"}
Oct 31 03:55:11.436935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.436892    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:11.446829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.446798    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:11.450105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.450078    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:11.551084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551046    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\") pod \"04e5b8ab-d33f-4af1-866a-e047b98261da\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") "
Oct 31 03:55:11.551309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551133    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\" (UniqueName: \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\") pod \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") "
Oct 31 03:55:11.551309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551205    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\") pod \"04e5b8ab-d33f-4af1-866a-e047b98261da\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") "
Oct 31 03:55:11.551309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551299    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\" (UniqueName: \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\") pod \"4634040b-5664-4610-90de-a2c62230035c\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") "
Oct 31 03:55:11.551490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551352    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\") pod \"4634040b-5664-4610-90de-a2c62230035c\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") "
Oct 31 03:55:11.551490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551380    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\") pod \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") "
Oct 31 03:55:11.551595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551510    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\") pod \"4634040b-5664-4610-90de-a2c62230035c\" (UID: \"4634040b-5664-4610-90de-a2c62230035c\") "
Oct 31 03:55:11.551699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551628    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\" (UniqueName: \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\") pod \"04e5b8ab-d33f-4af1-866a-e047b98261da\" (UID: \"04e5b8ab-d33f-4af1-866a-e047b98261da\") "
Oct 31 03:55:11.551760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551745    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\") pod \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\" (UID: \"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928\") "
Oct 31 03:55:11.551859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.551816    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca" pod "4634040b-5664-4610-90de-a2c62230035c" (UID: "4634040b-5664-4610-90de-a2c62230035c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:11.552108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.552076    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca" pod "04e5b8ab-d33f-4af1-866a-e047b98261da" (UID: "04e5b8ab-d33f-4af1-866a-e047b98261da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:11.552472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.552449    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca" pod "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" (UID: "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:11.552874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.552847    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.552979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.552880    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.552979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.552896    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4634040b-5664-4610-90de-a2c62230035c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.554572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.554549    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb" pod "04e5b8ab-d33f-4af1-866a-e047b98261da" (UID: "04e5b8ab-d33f-4af1-866a-e047b98261da"). InnerVolumeSpecName "kube-api-access-zp4zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:11.554767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.554587    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm" pod "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" (UID: "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928"). InnerVolumeSpecName "kube-api-access-fl5zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:11.554837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.554791    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca" pod "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" (UID: "4ad13573-fe2a-4b62-9e05-7a9c3a3a2928"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:11.554837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.554790    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca" pod "4634040b-5664-4610-90de-a2c62230035c" (UID: "4634040b-5664-4610-90de-a2c62230035c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:11.554920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.554877    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca" pod "04e5b8ab-d33f-4af1-866a-e047b98261da" (UID: "04e5b8ab-d33f-4af1-866a-e047b98261da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:11.555726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.555704    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79" pod "4634040b-5664-4610-90de-a2c62230035c" (UID: "4634040b-5664-4610-90de-a2c62230035c"). InnerVolumeSpecName "kube-api-access-8fb79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:11.655089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655055    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fb79\" (UniqueName: \"kubernetes.io/projected/4634040b-5664-4610-90de-a2c62230035c-kube-api-access-8fb79\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.655089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655086    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.655089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655095    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4634040b-5664-4610-90de-a2c62230035c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.655359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655105    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp4zb\" (UniqueName: \"kubernetes.io/projected/04e5b8ab-d33f-4af1-866a-e047b98261da-kube-api-access-zp4zb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.655359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655114    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04e5b8ab-d33f-4af1-866a-e047b98261da-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.655359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.655123    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl5zm\" (UniqueName: \"kubernetes.io/projected/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928-kube-api-access-fl5zm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:11.754961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.754928    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"
Oct 31 03:55:11.754961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.754957    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8" event={"ID":"04e5b8ab-d33f-4af1-866a-e047b98261da","Type":"ContainerDied","Data":"e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1"}
Oct 31 03:55:11.755585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.755011    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5c8f9fdc41f46d40248b1a32ede2de01de23ff0a7b031744115dbe9ecf5d6e1"
Oct 31 03:55:11.757144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.757110    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs" event={"ID":"4634040b-5664-4610-90de-a2c62230035c","Type":"ContainerDied","Data":"cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f"}
Oct 31 03:55:11.757292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.757165    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cadc5a1529a5e8dbb1f7170681a68848db21c7f6af4d40aa97922101fc86558f"
Oct 31 03:55:11.757292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.757119    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"
Oct 31 03:55:11.759706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.759375    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"
Oct 31 03:55:11.759706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.759374    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth" event={"ID":"4ad13573-fe2a-4b62-9e05-7a9c3a3a2928","Type":"ContainerDied","Data":"76719e8c3195d55edf5bd251c8be479ef95590f0f459451c71d94adcb7b414bd"}
Oct 31 03:55:11.759706 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:11.759501    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76719e8c3195d55edf5bd251c8be479ef95590f0f459451c71d94adcb7b414bd"
Oct 31 03:55:12.019438 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.019408    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:12.082036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.081944    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\") pod \"886ab726-69c6-41e0-9fd4-237790b9fee3\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") "
Oct 31 03:55:12.082234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.082090    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\") pod \"886ab726-69c6-41e0-9fd4-237790b9fee3\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") "
Oct 31 03:55:12.082234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.082185    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\" (UniqueName: \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\") pod \"886ab726-69c6-41e0-9fd4-237790b9fee3\" (UID: \"886ab726-69c6-41e0-9fd4-237790b9fee3\") "
Oct 31 03:55:12.082555 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.082524    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca" pod "886ab726-69c6-41e0-9fd4-237790b9fee3" (UID: "886ab726-69c6-41e0-9fd4-237790b9fee3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:12.083029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.083005    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:12.084720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.084671    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548" pod "886ab726-69c6-41e0-9fd4-237790b9fee3" (UID: "886ab726-69c6-41e0-9fd4-237790b9fee3"). InnerVolumeSpecName "kube-api-access-zq548". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:12.084911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.084898    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca" pod "886ab726-69c6-41e0-9fd4-237790b9fee3" (UID: "886ab726-69c6-41e0-9fd4-237790b9fee3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:12.184635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.184580    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/886ab726-69c6-41e0-9fd4-237790b9fee3-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:12.184635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.184632    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq548\" (UniqueName: \"kubernetes.io/projected/886ab726-69c6-41e0-9fd4-237790b9fee3-kube-api-access-zq548\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:12.789661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.789391    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"
Oct 31 03:55:12.789661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.789424    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d" event={"ID":"886ab726-69c6-41e0-9fd4-237790b9fee3","Type":"ContainerDied","Data":"74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c"}
Oct 31 03:55:12.789661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:12.789485    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74fce687375d23ade298669dfdc575beb00243576890524f98be6ceeac95c03c"
Oct 31 03:55:13.144838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:13.144735    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:55:13.145029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:13.144967    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy" containerID="cri-o://c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc" gracePeriod=30
Oct 31 03:55:13.150688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:13.150655    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:55:13.150935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:13.150901    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy" containerID="cri-o://466caa37a368efe74f63f992f94aa665dfa44864b0c019893b6213656432c24d" gracePeriod=30
Oct 31 03:55:15.477763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:15.477718    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:55:15.478199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:15.477954    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" containerID="cri-o://81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a" gracePeriod=30
Oct 31 03:55:16.381641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.380795    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"]
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383106    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383158    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383228    5098 cpu_manager.go:414] "containerMap: removing container" podUID="04e5b8ab-d33f-4af1-866a-e047b98261da" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383238    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04e5b8ab-d33f-4af1-866a-e047b98261da" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383253    5098 cpu_manager.go:414] "containerMap: removing container" podUID="886ab726-69c6-41e0-9fd4-237790b9fee3" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383263    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="886ab726-69c6-41e0-9fd4-237790b9fee3" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383298    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4634040b-5664-4610-90de-a2c62230035c" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383306    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4634040b-5664-4610-90de-a2c62230035c" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383826    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383861    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="04e5b8ab-d33f-4af1-866a-e047b98261da" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383873    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="886ab726-69c6-41e0-9fd4-237790b9fee3" containerName="revoke-keys"
Oct 31 03:55:16.385638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.383890    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4634040b-5664-4610-90de-a2c62230035c" containerName="revoke-keys"
Oct 31 03:55:16.442485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.440850    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"]
Oct 31 03:55:16.442485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.441001    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.461296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.461267    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:16.462445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.462412    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:55:16.616267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.616231    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.616704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.616402    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.616704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.616519    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4vqd\" (UniqueName: \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.718120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.718032    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.718284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.718145    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4vqd\" (UniqueName: \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.718362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.718348    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.718908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.718874    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.720883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.720851    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.730408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.730379    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4vqd\" (UniqueName: \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\") pod \"maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:16.783376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:16.783339    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:17.083842 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:17.083806    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50312768_d894_4d93_a05b_2bb3c89b907c.slice/crio-c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81 WatchSource:0}: Error finding container c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81: Status 404 returned error can't find the container with id c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81
Oct 31 03:55:17.084887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.084862    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"]
Oct 31 03:55:17.254767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.254728    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:55:17.270748 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.270716    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:55:17.512256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.512221    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:55:17.528111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.528078    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:55:17.908977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.908925    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s" event={"ID":"50312768-d894-4d93-a05b-2bb3c89b907c","Type":"ContainerStarted","Data":"62d3cea6e5efc13e5572809c96c4b03ad8b198a97c7b6479f94a2ff2a26287d2"}
Oct 31 03:55:17.909545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.909518    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s" event={"ID":"50312768-d894-4d93-a05b-2bb3c89b907c","Type":"ContainerStarted","Data":"c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81"}
Oct 31 03:55:17.993630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:17.991085    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s" podStartSLOduration=1.991060887 podStartE2EDuration="1.991060887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:17.959145519 +0000 UTC m=+2392.484365109" watchObservedRunningTime="2026-10-31 03:55:17.991060887 +0000 UTC m=+2392.516280476"
Oct 31 03:55:18.183042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.176910    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:55:18.848273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.848234    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:55:18.922560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.922524    5098 generic.go:356] "Generic (PLEG): container finished" podID="5e7c907e-d182-499e-903d-f57ff34e667f" containerID="c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc" exitCode=0
Oct 31 03:55:18.923038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.922563    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" event={"ID":"5e7c907e-d182-499e-903d-f57ff34e667f","Type":"ContainerDied","Data":"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"}
Oct 31 03:55:18.923038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.922594    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"
Oct 31 03:55:18.923038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.922700    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv" event={"ID":"5e7c907e-d182-499e-903d-f57ff34e667f","Type":"ContainerDied","Data":"bc6acd32a61a0a4b91c03953923a21a09c40f47830376ef7f87e14a64f9d6d2c"}
Oct 31 03:55:18.923038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.922750    5098 scope.go:122] "RemoveContainer" containerID="c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"
Oct 31 03:55:18.924471 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.924447    5098 generic.go:356] "Generic (PLEG): container finished" podID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerID="466caa37a368efe74f63f992f94aa665dfa44864b0c019893b6213656432c24d" exitCode=0
Oct 31 03:55:18.924722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.924617    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" event={"ID":"46b30cc3-de1e-4da8-90f5-01e43bb4ca55","Type":"ContainerDied","Data":"466caa37a368efe74f63f992f94aa665dfa44864b0c019893b6213656432c24d"}
Oct 31 03:55:18.924722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.924678    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x" event={"ID":"46b30cc3-de1e-4da8-90f5-01e43bb4ca55","Type":"ContainerDied","Data":"4fee762b542952c2c7543ebe07f3dc38a5571ed2eb76c6e2ba9c018f818e2d08"}
Oct 31 03:55:18.924722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.924703    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fee762b542952c2c7543ebe07f3dc38a5571ed2eb76c6e2ba9c018f818e2d08"
Oct 31 03:55:18.926662 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.926580    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:55:18.927079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.927058    5098 generic.go:356] "Generic (PLEG): container finished" podID="50312768-d894-4d93-a05b-2bb3c89b907c" containerID="62d3cea6e5efc13e5572809c96c4b03ad8b198a97c7b6479f94a2ff2a26287d2" exitCode=0
Oct 31 03:55:18.927226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.927212    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s" event={"ID":"50312768-d894-4d93-a05b-2bb3c89b907c","Type":"ContainerDied","Data":"62d3cea6e5efc13e5572809c96c4b03ad8b198a97c7b6479f94a2ff2a26287d2"}
Oct 31 03:55:18.936421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.936404    5098 scope.go:122] "RemoveContainer" containerID="c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"
Oct 31 03:55:18.936785 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:18.936757    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc\": container with ID starting with c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc not found: ID does not exist" containerID="c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"
Oct 31 03:55:18.936883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.936794    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc"} err="failed to get container status \"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc\": rpc error: code = NotFound desc = could not find container \"c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc\": container with ID starting with c09a923b28d4142980575e7f728b0247da50b5d1baff81b39411445c6ed5cdfc not found: ID does not exist"
Oct 31 03:55:18.982911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.982873    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.982937    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.982991    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983053    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.983166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983083    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983194    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983279    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983363    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983437    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983512    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.983585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983507    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983648    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983700    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983768    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983792    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983857    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.983987    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.984036    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\") pod \"5e7c907e-d182-499e-903d-f57ff34e667f\" (UID: \"5e7c907e-d182-499e-903d-f57ff34e667f\") "
Oct 31 03:55:18.984169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.984070    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\") pod \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\" (UID: \"46b30cc3-de1e-4da8-90f5-01e43bb4ca55\") "
Oct 31 03:55:18.985134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.985110    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:18.985233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.985138    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:18.987316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.985781    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.987316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.986056    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.987316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.986283    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.987316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.986581    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:18.987316 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.986869    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.987859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.987831    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:18.988936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.988914    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.989544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.989076    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:18.995129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.995088    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:18.995518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:18.995478    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:19.009720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.009671    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:55:19.011299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.011238    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:19.011709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.011663    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "kube-api-access-4hzgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:19.011994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.011953    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:19.012282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.012246    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo" pod "5e7c907e-d182-499e-903d-f57ff34e667f" (UID: "5e7c907e-d182-499e-903d-f57ff34e667f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:19.014551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.014513    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "kube-api-access-s4v6k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:19.016452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.016425    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo" pod "46b30cc3-de1e-4da8-90f5-01e43bb4ca55" (UID: "46b30cc3-de1e-4da8-90f5-01e43bb4ca55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087143    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087179    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087190    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087200    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087209    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087218    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e7c907e-d182-499e-903d-f57ff34e667f-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087228    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087240    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087253    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e7c907e-d182-499e-903d-f57ff34e667f-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087262    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087271    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087279    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087288    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087296    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hzgr\" (UniqueName: \"kubernetes.io/projected/5e7c907e-d182-499e-903d-f57ff34e667f-kube-api-access-4hzgr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087305    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e7c907e-d182-499e-903d-f57ff34e667f-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.087574 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.087316    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4v6k\" (UniqueName: \"kubernetes.io/projected/46b30cc3-de1e-4da8-90f5-01e43bb4ca55-kube-api-access-s4v6k\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:19.253682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.253638    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:55:19.313147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.313092    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-xlfxv"]
Oct 31 03:55:19.640530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.640495    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" path="/var/lib/kubelet/pods/5e7c907e-d182-499e-903d-f57ff34e667f/volumes"
Oct 31 03:55:19.942981 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.941979    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"
Oct 31 03:55:19.991091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:19.991017    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:55:20.001930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.001875    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-jmb6x"]
Oct 31 03:55:20.211582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.208263    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:55:20.401751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.382317    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-lgqnh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:55:20.401751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.382398    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:55:20.595077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.594977    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:20.660257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.660219    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\" (UniqueName: \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\") pod \"50312768-d894-4d93-a05b-2bb3c89b907c\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") "
Oct 31 03:55:20.660871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.660841    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\") pod \"50312768-d894-4d93-a05b-2bb3c89b907c\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") "
Oct 31 03:55:20.661024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.660940    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\") pod \"50312768-d894-4d93-a05b-2bb3c89b907c\" (UID: \"50312768-d894-4d93-a05b-2bb3c89b907c\") "
Oct 31 03:55:20.662030 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.661999    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca" pod "50312768-d894-4d93-a05b-2bb3c89b907c" (UID: "50312768-d894-4d93-a05b-2bb3c89b907c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:20.663943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.663916    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd" pod "50312768-d894-4d93-a05b-2bb3c89b907c" (UID: "50312768-d894-4d93-a05b-2bb3c89b907c"). InnerVolumeSpecName "kube-api-access-w4vqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:20.664141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.664111    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca" pod "50312768-d894-4d93-a05b-2bb3c89b907c" (UID: "50312768-d894-4d93-a05b-2bb3c89b907c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:20.763253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.763163    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.763253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.763195    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50312768-d894-4d93-a05b-2bb3c89b907c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.763253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.763207    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4vqd\" (UniqueName: \"kubernetes.io/projected/50312768-d894-4d93-a05b-2bb3c89b907c-kube-api-access-w4vqd\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.799046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.799016    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:55:20.863834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.863789    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.863977    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864006    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864087    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864122    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:20.864192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864131    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864203    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864284    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864365    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:20.864480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864377    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:20.864480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864382    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.864670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864538    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\") pod \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\" (UID: \"03c8bef5-4de6-46dd-9356-8cf4b282cb16\") "
Oct 31 03:55:20.865011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.864985    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:20.865208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865181    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:20.865521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865496    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.865689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865524    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.865689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865539    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.865689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865552    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.865689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.865565    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:20.867147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.867117    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:20.867352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.867327    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "kube-api-access-j2xwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:20.867575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.867558    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:20.867753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:20.867731    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo" pod "03c8bef5-4de6-46dd-9356-8cf4b282cb16" (UID: "03c8bef5-4de6-46dd-9356-8cf4b282cb16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:21.025282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.025041    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:21.025282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.025243    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2xwx\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-kube-api-access-j2xwx\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:21.025282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.025265    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:21.025282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.025281    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c8bef5-4de6-46dd-9356-8cf4b282cb16-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:21.043720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.043679    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"
Oct 31 03:55:21.044009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.043896    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s" event={"ID":"50312768-d894-4d93-a05b-2bb3c89b907c","Type":"ContainerDied","Data":"c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81"}
Oct 31 03:55:21.044140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.044085    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0e8dbe4c49cd214dd0cc8bdfc71dae67abf0a6c009b7c6f70d7628e70e76f81"
Oct 31 03:55:21.045983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.045950    5098 generic.go:356] "Generic (PLEG): container finished" podID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerID="81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a" exitCode=0
Oct 31 03:55:21.046182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.046162    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" event={"ID":"03c8bef5-4de6-46dd-9356-8cf4b282cb16","Type":"ContainerDied","Data":"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"}
Oct 31 03:55:21.046324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.046308    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh" event={"ID":"03c8bef5-4de6-46dd-9356-8cf4b282cb16","Type":"ContainerDied","Data":"ea7fcc9f9657c876bcc399baa97026512aa51432efdfcf05c414ad51062dd757"}
Oct 31 03:55:21.046453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.046422    5098 scope.go:122] "RemoveContainer" containerID="81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"
Oct 31 03:55:21.046742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.046723    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"
Oct 31 03:55:21.060754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.060733    5098 scope.go:122] "RemoveContainer" containerID="81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"
Oct 31 03:55:21.061061 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:21.061041    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a\": container with ID starting with 81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a not found: ID does not exist" containerID="81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"
Oct 31 03:55:21.061114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.061072    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a"} err="failed to get container status \"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a\": rpc error: code = NotFound desc = could not find container \"81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a\": container with ID starting with 81ba92e2f041ed2895daea7e2f2f4462b8bd0cebfd12afbea63a079bf851e45a not found: ID does not exist"
Oct 31 03:55:21.080530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.080504    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:55:21.087671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.087642    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-lgqnh"]
Oct 31 03:55:21.614889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.614853    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" path="/var/lib/kubelet/pods/03c8bef5-4de6-46dd-9356-8cf4b282cb16/volumes"
Oct 31 03:55:21.615487 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:21.615470    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" path="/var/lib/kubelet/pods/46b30cc3-de1e-4da8-90f5-01e43bb4ca55/volumes"
Oct 31 03:55:22.255482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.255445    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerName="payload-processing" containerID="cri-o://aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95" gracePeriod=25
Oct 31 03:55:22.272475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.272318    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerName="payload-pre-processing" containerID="cri-o://ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f" gracePeriod=25
Oct 31 03:55:22.416476 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:22.416446    5098 logging.go:55] [core] [Channel #8164 SubChannel #8166]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.235:50052", ServerName: "10.132.0.235:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.235:50052: connect: connection refused"
Oct 31 03:55:22.416692 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:22.416483    5098 logging.go:55] [core] [Channel #8165 SubChannel #8167]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.236:50052", ServerName: "10.132.0.236:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.236:50052: connect: connection refused"
Oct 31 03:55:22.512936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.512851    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing" containerID="cri-o://6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47" gracePeriod=25
Oct 31 03:55:22.532760 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.532708    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing" containerID="cri-o://e5ab75a1bbc975eb9af8d6b6653df9cd9e4effe94643540f09ebffde9e20716d" gracePeriod=25
Oct 31 03:55:22.608123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.607710    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:55:22.679421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.679385    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:55:22.756108 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.756069    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\") pod \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") "
Oct 31 03:55:22.756302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.756175    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\") pod \"078cad6a-6136-451f-9da0-9747a8b7dac4\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") "
Oct 31 03:55:22.756302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.756240    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\") pod \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\" (UID: \"ba63a01f-093e-4a6f-abb0-34c000a5b3ef\") "
Oct 31 03:55:22.756448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.756336    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\" (UniqueName: \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\") pod \"078cad6a-6136-451f-9da0-9747a8b7dac4\" (UID: \"078cad6a-6136-451f-9da0-9747a8b7dac4\") "
Oct 31 03:55:22.756562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.756538    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume" pod "ba63a01f-093e-4a6f-abb0-34c000a5b3ef" (UID: "ba63a01f-093e-4a6f-abb0-34c000a5b3ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:22.758236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.757575    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:22.759271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.759202    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume" pod "078cad6a-6136-451f-9da0-9747a8b7dac4" (UID: "078cad6a-6136-451f-9da0-9747a8b7dac4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:22.759403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.759380    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk" pod "ba63a01f-093e-4a6f-abb0-34c000a5b3ef" (UID: "ba63a01f-093e-4a6f-abb0-34c000a5b3ef"). InnerVolumeSpecName "kube-api-access-7ctbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:22.759718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.759686    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz" pod "078cad6a-6136-451f-9da0-9747a8b7dac4" (UID: "078cad6a-6136-451f-9da0-9747a8b7dac4"). InnerVolumeSpecName "kube-api-access-vswmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:22.860936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.859530    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/078cad6a-6136-451f-9da0-9747a8b7dac4-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:22.860936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.859565    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ctbk\" (UniqueName: \"kubernetes.io/projected/ba63a01f-093e-4a6f-abb0-34c000a5b3ef-kube-api-access-7ctbk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:22.860936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.859580    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vswmz\" (UniqueName: \"kubernetes.io/projected/078cad6a-6136-451f-9da0-9747a8b7dac4-kube-api-access-vswmz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:22.992785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.992745    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:55:22.992993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:22.992962    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy" containerID="cri-o://03debde827ac5a10e924272ea14f621281a735293ddd56c958d5ec3f19d94115" gracePeriod=30
Oct 31 03:55:23.093356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.093324    5098 generic.go:356] "Generic (PLEG): container finished" podID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerID="ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f" exitCode=0
Oct 31 03:55:23.093527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.093373    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"
Oct 31 03:55:23.093527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.093403    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" event={"ID":"ba63a01f-093e-4a6f-abb0-34c000a5b3ef","Type":"ContainerDied","Data":"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"}
Oct 31 03:55:23.093527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.093456    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" event={"ID":"ba63a01f-093e-4a6f-abb0-34c000a5b3ef","Type":"ContainerDied","Data":"51957bab73926a96d477c87c637d629b22e52073b7b1da221c8f8f6151282f2d"}
Oct 31 03:55:23.093527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.093488    5098 scope.go:122] "RemoveContainer" containerID="ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"
Oct 31 03:55:23.095290 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.095262    5098 generic.go:356] "Generic (PLEG): container finished" podID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerID="aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95" exitCode=0
Oct 31 03:55:23.095408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.095323    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"
Oct 31 03:55:23.095408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.095343    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" event={"ID":"078cad6a-6136-451f-9da0-9747a8b7dac4","Type":"ContainerDied","Data":"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"}
Oct 31 03:55:23.095408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.095394    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" event={"ID":"078cad6a-6136-451f-9da0-9747a8b7dac4","Type":"ContainerDied","Data":"2a46af755e068eb55e1d5c7db4b1eff65e0154d830358bc85949926739965551"}
Oct 31 03:55:23.119167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.119035    5098 scope.go:122] "RemoveContainer" containerID="ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"
Oct 31 03:55:23.119469 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:23.119441    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f\": container with ID starting with ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f not found: ID does not exist" containerID="ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"
Oct 31 03:55:23.119599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.119478    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f"} err="failed to get container status \"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f\": rpc error: code = NotFound desc = could not find container \"ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f\": container with ID starting with ec83d8b390c0180dd0caf381f55477ab3c0c52e149d39ce5c88106d8687edb9f not found: ID does not exist"
Oct 31 03:55:23.119599 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.119503    5098 scope.go:122] "RemoveContainer" containerID="aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"
Oct 31 03:55:23.131953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.131930    5098 scope.go:122] "RemoveContainer" containerID="aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"
Oct 31 03:55:23.132280 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:23.132259    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95\": container with ID starting with aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95 not found: ID does not exist" containerID="aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"
Oct 31 03:55:23.132352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.132294    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95"} err="failed to get container status \"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95\": rpc error: code = NotFound desc = could not find container \"aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95\": container with ID starting with aeadc466dde02ee4737564dbfd0f5b8291986ef2f640bf1b7bc3b5b4a9c4ae95 not found: ID does not exist"
Oct 31 03:55:23.140279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.140245    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:55:23.145584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.145553    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx"]
Oct 31 03:55:23.161543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.161494    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:55:23.172394 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.172336    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg"]
Oct 31 03:55:23.177907 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.177842    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" podUID="46903010-6eb1-4d77-b01d-7dd67664d8ac" containerName="payload-pre-processing" containerID="cri-o://1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376" gracePeriod=25
Oct 31 03:55:23.415912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.415656    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-b8fd22-7847b845cf-hkdrx" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.236:50052\" within 1s: context deadline exceeded"
Oct 31 03:55:23.415912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.415792    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-b8fd22-696dd68f69-zcgdg" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.235:50052\" within 1s: context deadline exceeded"
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.710848    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.839291    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" path="/var/lib/kubelet/pods/078cad6a-6136-451f-9da0-9747a8b7dac4/volumes"
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.842705    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" path="/var/lib/kubelet/pods/ba63a01f-093e-4a6f-abb0-34c000a5b3ef/volumes"
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.881102    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\") pod \"46903010-6eb1-4d77-b01d-7dd67664d8ac\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") "
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.881179    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\" (UniqueName: \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\") pod \"46903010-6eb1-4d77-b01d-7dd67664d8ac\" (UID: \"46903010-6eb1-4d77-b01d-7dd67664d8ac\") "
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.882136    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume" pod "46903010-6eb1-4d77-b01d-7dd67664d8ac" (UID: "46903010-6eb1-4d77-b01d-7dd67664d8ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.882531    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46903010-6eb1-4d77-b01d-7dd67664d8ac-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:23.895913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.884788    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62" pod "46903010-6eb1-4d77-b01d-7dd67664d8ac" (UID: "46903010-6eb1-4d77-b01d-7dd67664d8ac"). InnerVolumeSpecName "kube-api-access-mfv62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:23.973509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.973475    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"]
Oct 31 03:55:23.974454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.974348    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:55:23.982086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.981713    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-b8fd22-4wxvs"]
Oct 31 03:55:23.984156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:23.984121    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfv62\" (UniqueName: \"kubernetes.io/projected/46903010-6eb1-4d77-b01d-7dd67664d8ac-kube-api-access-mfv62\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:24.024310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.024231    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" podUID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" containerName="payload-pre-processing" containerID="cri-o://a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10" gracePeriod=25
Oct 31 03:55:24.109910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.109872    5098 generic.go:356] "Generic (PLEG): container finished" podID="46903010-6eb1-4d77-b01d-7dd67664d8ac" containerID="1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376" exitCode=0
Oct 31 03:55:24.110106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.109934    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" event={"ID":"46903010-6eb1-4d77-b01d-7dd67664d8ac","Type":"ContainerDied","Data":"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"}
Oct 31 03:55:24.110106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.109943    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"
Oct 31 03:55:24.110106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.110000    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw" event={"ID":"46903010-6eb1-4d77-b01d-7dd67664d8ac","Type":"ContainerDied","Data":"2546377de1ee72a9d30f458ff9ff3911383feeaad435905cf9f782c55bb22ec4"}
Oct 31 03:55:24.110106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.110022    5098 scope.go:122] "RemoveContainer" containerID="1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"
Oct 31 03:55:24.141150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.141117    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:55:24.144588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.144561    5098 scope.go:122] "RemoveContainer" containerID="1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"
Oct 31 03:55:24.145009 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:24.144984    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376\": container with ID starting with 1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376 not found: ID does not exist" containerID="1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"
Oct 31 03:55:24.145106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.145023    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376"} err="failed to get container status \"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376\": rpc error: code = NotFound desc = could not find container \"1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376\": container with ID starting with 1d5a5353e444337a9564213a87534d29c688e816fe0389402ba90400b5f37376 not found: ID does not exist"
Oct 31 03:55:24.146741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.146688    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-d4145b-b97d5fb8c-799cw"]
Oct 31 03:55:24.341593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.341565    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:55:24.492118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.492034    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\" (UniqueName: \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\") pod \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") "
Oct 31 03:55:24.492118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.492112    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\") pod \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\" (UID: \"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d\") "
Oct 31 03:55:24.492563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.492477    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume" pod "1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" (UID: "1f633e7a-8a73-4ed1-a75f-aa9725c45d5d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:24.492983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.492960    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:24.494460 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.494431    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726" pod "1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" (UID: "1f633e7a-8a73-4ed1-a75f-aa9725c45d5d"). InnerVolumeSpecName "kube-api-access-mg726". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:24.594716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.594680    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg726\" (UniqueName: \"kubernetes.io/projected/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d-kube-api-access-mg726\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:24.671597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.671561    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:55:24.671811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:24.671788    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery" containerID="cri-o://f2f279491312e817c79328be6d5ed1a8166f9cde71c03caf09094b2b05adb4bd" gracePeriod=30
Oct 31 03:55:25.064042 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.063947    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"]
Oct 31 03:55:25.077485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.077448    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0736dc06-zt6pf"]
Oct 31 03:55:25.082284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.082245    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"]
Oct 31 03:55:25.089134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.089095    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-d4145b-jnrth"]
Oct 31 03:55:25.151093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.151052    5098 generic.go:356] "Generic (PLEG): container finished" podID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerID="f2f279491312e817c79328be6d5ed1a8166f9cde71c03caf09094b2b05adb4bd" exitCode=0
Oct 31 03:55:25.151431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.151406    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" event={"ID":"de93c07b-3571-4b99-a12f-8e40941f28c2","Type":"ContainerDied","Data":"f2f279491312e817c79328be6d5ed1a8166f9cde71c03caf09094b2b05adb4bd"}
Oct 31 03:55:25.154841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.154809    5098 generic.go:356] "Generic (PLEG): container finished" podID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" containerID="a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10" exitCode=0
Oct 31 03:55:25.155285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.155260    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" event={"ID":"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d","Type":"ContainerDied","Data":"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"}
Oct 31 03:55:25.155463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.155448    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk" event={"ID":"1f633e7a-8a73-4ed1-a75f-aa9725c45d5d","Type":"ContainerDied","Data":"ac8aecdc66d3db3cc092805b65ec09e0fc0ca29b21da691c662844af72c3dbd3"}
Oct 31 03:55:25.155580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.155568    5098 scope.go:122] "RemoveContainer" containerID="a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"
Oct 31 03:55:25.155821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.155805    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"
Oct 31 03:55:25.313860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.313832    5098 scope.go:122] "RemoveContainer" containerID="a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"
Oct 31 03:55:25.314455 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:25.314399    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10\": container with ID starting with a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10 not found: ID does not exist" containerID="a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"
Oct 31 03:55:25.315026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.314597    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10"} err="failed to get container status \"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10\": rpc error: code = NotFound desc = could not find container \"a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10\": container with ID starting with a1d7f9c10c0fb7e3bcdf6888022523180ecbc89704c9b6da4e3974aae51fcb10 not found: ID does not exist"
Oct 31 03:55:25.316998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.316980    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:55:25.327084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.324216    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-afb8be-6f8b5455f9-zgdsk"]
Oct 31 03:55:25.367813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.367785    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.566797    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.566936    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.566972    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.567132    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.567266    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.567320    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.567750 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.567363    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\") pod \"de93c07b-3571-4b99-a12f-8e40941f28c2\" (UID: \"de93c07b-3571-4b99-a12f-8e40941f28c2\") "
Oct 31 03:55:25.580975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.578333    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:25.592270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.588220    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:55:25.592270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.591320    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:25.592834 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.592662    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:25.593443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.593413    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:25.599400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.599358    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "kube-api-access-vnvbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:25.600041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.600016    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:25.600181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.600124    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig" pod "de93c07b-3571-4b99-a12f-8e40941f28c2" (UID: "de93c07b-3571-4b99-a12f-8e40941f28c2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:25.632531 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.632446    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" path="/var/lib/kubelet/pods/1f633e7a-8a73-4ed1-a75f-aa9725c45d5d/volumes"
Oct 31 03:55:25.633366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.633343    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4634040b-5664-4610-90de-a2c62230035c" path="/var/lib/kubelet/pods/4634040b-5664-4610-90de-a2c62230035c/volumes"
Oct 31 03:55:25.634632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.633862    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46903010-6eb1-4d77-b01d-7dd67664d8ac" path="/var/lib/kubelet/pods/46903010-6eb1-4d77-b01d-7dd67664d8ac/volumes"
Oct 31 03:55:25.635052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.635035    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad13573-fe2a-4b62-9e05-7a9c3a3a2928" path="/var/lib/kubelet/pods/4ad13573-fe2a-4b62-9e05-7a9c3a3a2928/volumes"
Oct 31 03:55:25.635520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.635503    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1d3d7d-10af-4086-86a0-4222364786f5" path="/var/lib/kubelet/pods/bc1d3d7d-10af-4086-86a0-4222364786f5/volumes"
Oct 31 03:55:25.676265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676231    5098 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-cacerts\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676265 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676264    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnvbv\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-kube-api-access-vnvbv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676275    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-dns-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676284    5098 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-kubeconfig\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676296    5098 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de93c07b-3571-4b99-a12f-8e40941f28c2-local-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676329    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-csr-ca-configmap\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.676461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.676338    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de93c07b-3571-4b99-a12f-8e40941f28c2-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:25.981704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:25.967541    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:55:26.227984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:26.227955    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"
Oct 31 03:55:26.227984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:26.227959    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" event={"ID":"de93c07b-3571-4b99-a12f-8e40941f28c2","Type":"ContainerDied","Data":"1b69b32eecc83054dff71a1a056eda3e4cc0110526f039ecbdad8ad104ee74e1"}
Oct 31 03:55:26.228266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:26.228052    5098 scope.go:122] "RemoveContainer" containerID="f2f279491312e817c79328be6d5ed1a8166f9cde71c03caf09094b2b05adb4bd"
Oct 31 03:55:26.255575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:26.255543    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:55:26.265633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:26.262516    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8"]
Oct 31 03:55:27.244157 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244126    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"]
Oct 31 03:55:27.244939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244923    5098 cpu_manager.go:414] "containerMap: removing container" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerName="payload-processing"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244940    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerName="payload-processing"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244955    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerName="payload-pre-processing"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244960    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerName="payload-pre-processing"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244971    5098 cpu_manager.go:414] "containerMap: removing container" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244976    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244985    5098 cpu_manager.go:414] "containerMap: removing container" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy"
Oct 31 03:55:27.245007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.244990    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245019    5098 cpu_manager.go:414] "containerMap: removing container" podUID="46903010-6eb1-4d77-b01d-7dd67664d8ac" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245026    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46903010-6eb1-4d77-b01d-7dd67664d8ac" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245038    5098 cpu_manager.go:414] "containerMap: removing container" podUID="50312768-d894-4d93-a05b-2bb3c89b907c" containerName="revoke-keys"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245043    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50312768-d894-4d93-a05b-2bb3c89b907c" containerName="revoke-keys"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245056    5098 cpu_manager.go:414] "containerMap: removing container" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245060    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245072    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245076    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245082    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245088    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245195    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c8bef5-4de6-46dd-9356-8cf4b282cb16" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245203    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7c907e-d182-499e-903d-f57ff34e667f" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245211    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f633e7a-8a73-4ed1-a75f-aa9725c45d5d" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245218    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="078cad6a-6136-451f-9da0-9747a8b7dac4" containerName="payload-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245224    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba63a01f-093e-4a6f-abb0-34c000a5b3ef" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245230    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245237    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b30cc3-de1e-4da8-90f5-01e43bb4ca55" containerName="istio-proxy"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245248    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="46903010-6eb1-4d77-b01d-7dd67664d8ac" containerName="payload-pre-processing"
Oct 31 03:55:27.245276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.245255    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="50312768-d894-4d93-a05b-2bb3c89b907c" containerName="revoke-keys"
Oct 31 03:55:27.250740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.250715    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.253956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.253926    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.254110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.253968    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.254332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.254292    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz4v8\" (UniqueName: \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.256504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.256474    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:55:27.256757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.256576    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:27.261766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.261721    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"]
Oct 31 03:55:27.356013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.355972    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz4v8\" (UniqueName: \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.356223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.356119    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.356223 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.356144    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.356902 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.356871    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.358778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.358753    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.366217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.366180    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz4v8\" (UniqueName: \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\") pod \"maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.571581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.571546    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:27.623004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.622956    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" path="/var/lib/kubelet/pods/de93c07b-3571-4b99-a12f-8e40941f28c2/volumes"
Oct 31 03:55:27.882686 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:27.882652    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25cdffa1_b007_4a78_b2eb_baf53fd39cfb.slice/crio-69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0 WatchSource:0}: Error finding container 69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0: Status 404 returned error can't find the container with id 69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0
Oct 31 03:55:27.883220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:27.883110    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"]
Oct 31 03:55:28.309127 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.309081    5098 generic.go:356] "Generic (PLEG): container finished" podID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerID="03debde827ac5a10e924272ea14f621281a735293ddd56c958d5ec3f19d94115" exitCode=0
Oct 31 03:55:28.309698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.309295    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerDied","Data":"03debde827ac5a10e924272ea14f621281a735293ddd56c958d5ec3f19d94115"}
Oct 31 03:55:28.309698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.309380    5098 scope.go:122] "RemoveContainer" containerID="72c7d4b7c99a49b73bb9a51a00de7412033631a1587e09eccb8fb97d8d5410fa"
Oct 31 03:55:28.334358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.334306    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn" event={"ID":"25cdffa1-b007-4a78-b2eb-baf53fd39cfb","Type":"ContainerStarted","Data":"69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0"}
Oct 31 03:55:28.747757 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.747730    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:55:28.803515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803426    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.803515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803511    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.803906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803541    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.803906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803751    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.803906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803820    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.804084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.803996    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.804084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804047    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.804084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804055    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:28.804084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804078    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.804305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804083    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:28.804305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804166    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\") pod \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\" (UID: \"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d\") "
Oct 31 03:55:28.804440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804354    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:28.804581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.804556    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:28.805097 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.805068    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:28.805197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.805138    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.805197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.805161    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.805197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.805176    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.805197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.805190    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.806191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.806164    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:28.806808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.806785    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "kube-api-access-j9z4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:28.807008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.806984    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:28.807105 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.807049    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo" pod "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" (UID: "02e137cd-412b-41c1-a5fa-fbbf9d47ef8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:28.907339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.907299    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.907339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.907334    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.907577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.907353    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.907577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.907366    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:28.907577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:28.907378    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9z4t\" (UniqueName: \"kubernetes.io/projected/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d-kube-api-access-j9z4t\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:29.347806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.347773    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"
Oct 31 03:55:29.348346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.347800    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj" event={"ID":"02e137cd-412b-41c1-a5fa-fbbf9d47ef8d","Type":"ContainerDied","Data":"3b4e3595a2c9949ba9fbe844f9dce319047b7c989e01af8345056637da2c9f44"}
Oct 31 03:55:29.348346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.347871    5098 scope.go:122] "RemoveContainer" containerID="03debde827ac5a10e924272ea14f621281a735293ddd56c958d5ec3f19d94115"
Oct 31 03:55:29.350226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.350198    5098 generic.go:356] "Generic (PLEG): container finished" podID="25cdffa1-b007-4a78-b2eb-baf53fd39cfb" containerID="a03e8ef459dbdb5f0077a29210c81b8fc182f1fa4f128f2cc8458383508e30df" exitCode=0
Oct 31 03:55:29.350360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.350325    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn" event={"ID":"25cdffa1-b007-4a78-b2eb-baf53fd39cfb","Type":"ContainerDied","Data":"a03e8ef459dbdb5f0077a29210c81b8fc182f1fa4f128f2cc8458383508e30df"}
Oct 31 03:55:29.394732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.394694    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:55:29.402639 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.401787    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-lqwcj"]
Oct 31 03:55:29.616411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:29.616307    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" path="/var/lib/kubelet/pods/02e137cd-412b-41c1-a5fa-fbbf9d47ef8d/volumes"
Oct 31 03:55:30.336241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.336193    5098 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-zl6d8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:55:30.336430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.336263    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-zl6d8" podUID="de93c07b-3571-4b99-a12f-8e40941f28c2" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.67:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:55:30.547989 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.547960    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:30.640079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.639990    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\" (UniqueName: \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\") pod \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") "
Oct 31 03:55:30.640255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.640235    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\") pod \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") "
Oct 31 03:55:30.640506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.640489    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\") pod \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\" (UID: \"25cdffa1-b007-4a78-b2eb-baf53fd39cfb\") "
Oct 31 03:55:30.640842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.640818    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca" pod "25cdffa1-b007-4a78-b2eb-baf53fd39cfb" (UID: "25cdffa1-b007-4a78-b2eb-baf53fd39cfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:30.641186 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.641167    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:30.642326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.642296    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca" pod "25cdffa1-b007-4a78-b2eb-baf53fd39cfb" (UID: "25cdffa1-b007-4a78-b2eb-baf53fd39cfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:30.642450 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.642344    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8" pod "25cdffa1-b007-4a78-b2eb-baf53fd39cfb" (UID: "25cdffa1-b007-4a78-b2eb-baf53fd39cfb"). InnerVolumeSpecName "kube-api-access-jz4v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:30.742912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.742866    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:30.742912 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:30.742901    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz4v8\" (UniqueName: \"kubernetes.io/projected/25cdffa1-b007-4a78-b2eb-baf53fd39cfb-kube-api-access-jz4v8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:31.059046 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:31.059007    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:55:31.373240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:31.373150    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"
Oct 31 03:55:31.373240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:31.373189    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn" event={"ID":"25cdffa1-b007-4a78-b2eb-baf53fd39cfb","Type":"ContainerDied","Data":"69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0"}
Oct 31 03:55:31.373498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:31.373242    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69ca7914c13119d9548ac6158368321ed61eb459547b584f32d0f3925c0cedf0"
Oct 31 03:55:32.889291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:32.889256    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:55:32.903816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:32.903790    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 03:55:33.424110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.424068    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:55:33.424372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.424343    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" podUID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" containerName="maas-api" containerID="cri-o://3d7abdfd3b2a2bb9bf5554f800efe76bc513cfdd2939f31a93d37fb54e22451e" gracePeriod=30
Oct 31 03:55:33.565006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.564973    5098 generic.go:356] "Generic (PLEG): container finished" podID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" containerID="3d7abdfd3b2a2bb9bf5554f800efe76bc513cfdd2939f31a93d37fb54e22451e" exitCode=0
Oct 31 03:55:33.565165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.565049    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" event={"ID":"9e5c3968-8757-492b-88c5-f01a4c8db0dc","Type":"ContainerDied","Data":"3d7abdfd3b2a2bb9bf5554f800efe76bc513cfdd2939f31a93d37fb54e22451e"}
Oct 31 03:55:33.953226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.950269    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:55:33.970167 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.970132    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:55:33.973234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973202    5098 cpu_manager.go:414] "containerMap: removing container" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:33.973424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973247    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:33.973424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973373    5098 cpu_manager.go:414] "containerMap: removing container" podUID="25cdffa1-b007-4a78-b2eb-baf53fd39cfb" containerName="revoke-keys"
Oct 31 03:55:33.973424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973382    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25cdffa1-b007-4a78-b2eb-baf53fd39cfb" containerName="revoke-keys"
Oct 31 03:55:33.973730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973674    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="25cdffa1-b007-4a78-b2eb-baf53fd39cfb" containerName="revoke-keys"
Oct 31 03:55:33.973730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973713    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:33.973840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.973753    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:33.982439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.982209    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:55:33.982439 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.982328    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.001633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:33.990205    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-39e8cf-openshift-default-dockercfg-k2rhx\""
Oct 31 03:55:34.009732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.006910    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"]
Oct 31 03:55:34.172249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.150707    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5d9b6-8mmk8"]
Oct 31 03:55:34.172249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.152507    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.172249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.152691    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.266632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.265377    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.265464    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.284141    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.284314    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.284434    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wvw9\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.284736    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.298638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.284836    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.563815    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.563891    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.563943    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.563995    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.564080    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.564142    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wvw9\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.564311    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.564432    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.564499    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.565078    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.565360    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.565624    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.566164    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.581139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.566924    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.599628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.593005    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:55:34.626678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.617369    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.626678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.625299    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wvw9\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.635858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.635090    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq" event={"ID":"9e5c3968-8757-492b-88c5-f01a4c8db0dc","Type":"ContainerDied","Data":"62d9306739c27b1dc88195a8bf0b9975c5bea08cf2b694fb84485f8467220bf5"}
Oct 31 03:55:34.635858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.635165    5098 scope.go:122] "RemoveContainer" containerID="3d7abdfd3b2a2bb9bf5554f800efe76bc513cfdd2939f31a93d37fb54e22451e"
Oct 31 03:55:34.635858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.635381    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"
Oct 31 03:55:34.643408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.641898    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.650879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.650809    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\") pod \"e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") " pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.707807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.707766    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:34.767524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.767483    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\") pod \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") "
Oct 31 03:55:34.767759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.767737    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\") pod \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") "
Oct 31 03:55:34.768442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.767980    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\") pod \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") "
Oct 31 03:55:34.768442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.768126    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca" pod "9e5c3968-8757-492b-88c5-f01a4c8db0dc" (UID: "9e5c3968-8757-492b-88c5-f01a4c8db0dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:34.768442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.768130    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\" (UniqueName: \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\") pod \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\" (UID: \"9e5c3968-8757-492b-88c5-f01a4c8db0dc\") "
Oct 31 03:55:34.769136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.769111    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5c3968-8757-492b-88c5-f01a4c8db0dc-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:34.777024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.776991    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls" pod "9e5c3968-8757-492b-88c5-f01a4c8db0dc" (UID: "9e5c3968-8757-492b-88c5-f01a4c8db0dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:34.777300 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.777026    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg" pod "9e5c3968-8757-492b-88c5-f01a4c8db0dc" (UID: "9e5c3968-8757-492b-88c5-f01a4c8db0dc"). InnerVolumeSpecName "kube-api-access-j6vlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:34.779445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.779399    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs" pod "9e5c3968-8757-492b-88c5-f01a4c8db0dc" (UID: "9e5c3968-8757-492b-88c5-f01a4c8db0dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:34.871039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.870954    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:34.871039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.870984    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9e5c3968-8757-492b-88c5-f01a4c8db0dc-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:34.871039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.870994    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6vlg\" (UniqueName: \"kubernetes.io/projected/9e5c3968-8757-492b-88c5-f01a4c8db0dc-kube-api-access-j6vlg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:34.965342 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.965290    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:55:34.970501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:34.970468    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a5d9b6-5686d88fbf-hvfzq"]
Oct 31 03:55:35.007920 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:35.007872    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2df149_41c5_41c9_aeaf_2bd567db9cf0.slice/crio-e8ca9a0d4213336ee43e993e0396f321177ba5de70717f17fbea6914ad6fe747 WatchSource:0}: Error finding container e8ca9a0d4213336ee43e993e0396f321177ba5de70717f17fbea6914ad6fe747: Status 404 returned error can't find the container with id e8ca9a0d4213336ee43e993e0396f321177ba5de70717f17fbea6914ad6fe747
Oct 31 03:55:35.008174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.008146    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:55:35.011152 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.011119    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:35.011252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.011199    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:35.011252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.011231    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:35.346896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.346843    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" podUID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" containerName="main" containerID="cri-o://d383f31808b9c5a86b45296e33f462687f9d4e144b04ac44f1e42da7345a660b" gracePeriod=45
Oct 31 03:55:35.630069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.629987    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04e5b8ab-d33f-4af1-866a-e047b98261da" path="/var/lib/kubelet/pods/04e5b8ab-d33f-4af1-866a-e047b98261da/volumes"
Oct 31 03:55:35.630718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.630693    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" path="/var/lib/kubelet/pods/9e5c3968-8757-492b-88c5-f01a4c8db0dc/volumes"
Oct 31 03:55:35.646797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.646761    5098 generic.go:356] "Generic (PLEG): container finished" podID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" containerID="d383f31808b9c5a86b45296e33f462687f9d4e144b04ac44f1e42da7345a660b" exitCode=0
Oct 31 03:55:35.646959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.646843    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" event={"ID":"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114","Type":"ContainerDied","Data":"d383f31808b9c5a86b45296e33f462687f9d4e144b04ac44f1e42da7345a660b"}
Oct 31 03:55:35.646959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.646903    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl" event={"ID":"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114","Type":"ContainerDied","Data":"664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3"}
Oct 31 03:55:35.646959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.646920    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="664f98094db4e0db35ee4e6ca0a338fb67e8d5e8b0eed9f7cb1dad9b487cbcc3"
Oct 31 03:55:35.650159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.650122    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerStarted","Data":"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"}
Oct 31 03:55:35.650286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.650176    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerStarted","Data":"e8ca9a0d4213336ee43e993e0396f321177ba5de70717f17fbea6914ad6fe747"}
Oct 31 03:55:35.675360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.675269    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:55:35.677699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.677652    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podStartSLOduration=2.6776373060000003 podStartE2EDuration="2.677637306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:35.675020786 +0000 UTC m=+2410.200240375" watchObservedRunningTime="2026-10-31 03:55:35.677637306 +0000 UTC m=+2410.202856895"
Oct 31 03:55:35.708656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.708588    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:35.787370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787336    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\" (UniqueName: \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787385    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\" (UniqueName: \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787428    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787533    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787572    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787742    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\") pod \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\" (UID: \"37f143ee-1fd5-4aa1-ac2e-bcd0b3741114\") "
Oct 31 03:55:35.787909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787807    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:35.787909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787817    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:35.788085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.787958    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:35.788754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.788711    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.788754 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.788743    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.788961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.788760    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.790106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.790077    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "kube-api-access-zsrcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:35.790106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.790087    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:35.790261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.790120    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs" pod "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" (UID: "37f143ee-1fd5-4aa1-ac2e-bcd0b3741114"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:35.890681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.890550    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.890681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.890588    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrcp\" (UniqueName: \"kubernetes.io/projected/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-kube-api-access-zsrcp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.890681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.890602    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:35.946883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:35.946841    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:55:36.027221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.027170    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"]
Oct 31 03:55:36.032241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.032204    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b35f400-9vgsl"]
Oct 31 03:55:36.565007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.564968    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"]
Oct 31 03:55:36.570553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.570511    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-afb8be-26k7d"]
Oct 31 03:55:36.803641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.748557    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"
Oct 31 03:55:36.963203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:36.949411    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:37.006911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.006872    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:55:37.011666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.011628    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d04aad/auth-test-model-d04aad-kserve-5fdcd84bd5-frqvl"]
Oct 31 03:55:37.364033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.363995    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:55:37.381184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.381152    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:55:37.614161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.614066    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" path="/var/lib/kubelet/pods/37f143ee-1fd5-4aa1-ac2e-bcd0b3741114/volumes"
Oct 31 03:55:37.614698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.614678    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="886ab726-69c6-41e0-9fd4-237790b9fee3" path="/var/lib/kubelet/pods/886ab726-69c6-41e0-9fd4-237790b9fee3/volumes"
Oct 31 03:55:37.615292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.615267    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9390b98-efa3-449f-a5e1-61794ca09c06" path="/var/lib/kubelet/pods/f9390b98-efa3-449f-a5e1-61794ca09c06/volumes"
Oct 31 03:55:37.800632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.799854    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:37.824634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:37.819084    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:55:39.406146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.406107    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:55:39.407182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407157    5098 cpu_manager.go:414] "containerMap: removing container" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:39.407311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407184    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02e137cd-412b-41c1-a5fa-fbbf9d47ef8d" containerName="istio-proxy"
Oct 31 03:55:39.407311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407210    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" containerName="maas-api"
Oct 31 03:55:39.407311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407217    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" containerName="maas-api"
Oct 31 03:55:39.407311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407295    5098 cpu_manager.go:414] "containerMap: removing container" podUID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" containerName="main"
Oct 31 03:55:39.407311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407303    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" containerName="main"
Oct 31 03:55:39.407587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407497    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="37f143ee-1fd5-4aa1-ac2e-bcd0b3741114" containerName="main"
Oct 31 03:55:39.407587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.407520    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5c3968-8757-492b-88c5-f01a4c8db0dc" containerName="maas-api"
Oct 31 03:55:39.412883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.412857    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.418667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.418596    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-39e8cf\""
Oct 31 03:55:39.419880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.419842    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-39e8cf-dockercfg-2rzjd\""
Oct 31 03:55:39.427149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.427112    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:55:39.457009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.456971    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnztz\" (UniqueName: \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.457204 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.457099    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.473238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.473191    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:55:39.493702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.479845    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.495565 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.494921    5098 scope.go:122] "RemoveContainer" containerID="d383f31808b9c5a86b45296e33f462687f9d4e144b04ac44f1e42da7345a660b"
Oct 31 03:55:39.802655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.791414    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnztz\" (UniqueName: \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.803560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.803521    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gscj4\" (UniqueName: \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.803731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.803690    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.803939 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.803914    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.805277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.805247    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.834627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.832187    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnztz\" (UniqueName: \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\") pod \"payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:39.839418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.839308    5098 scope.go:122] "RemoveContainer" containerID="466caa37a368efe74f63f992f94aa665dfa44864b0c019893b6213656432c24d"
Oct 31 03:55:39.850072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.850041    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:55:39.905392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.905343    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.906087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.905761    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gscj4\" (UniqueName: \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.906253 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.906165    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:39.914592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:39.914566    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gscj4\" (UniqueName: \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\") pod \"payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:40.055652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.055539    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:40.117192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.117152    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:40.356221 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:40.347923    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40d382c5_1a49_46d8_93d1_2cacdae68564.slice/crio-b257709a151dd919422863eecac68fa34daab08b41691e70237be546e0fe9ac0 WatchSource:0}: Error finding container b257709a151dd919422863eecac68fa34daab08b41691e70237be546e0fe9ac0: Status 404 returned error can't find the container with id b257709a151dd919422863eecac68fa34daab08b41691e70237be546e0fe9ac0
Oct 31 03:55:40.356221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.348843    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:55:40.429559 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:40.429529    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60bb8f8e_9ab4_46ca_a1a6_ddf0a8cdc01a.slice/crio-0ee73b7775c18f11f2e96961665618ce32f657e99aecb131574de8eee0591378 WatchSource:0}: Error finding container 0ee73b7775c18f11f2e96961665618ce32f657e99aecb131574de8eee0591378: Status 404 returned error can't find the container with id 0ee73b7775c18f11f2e96961665618ce32f657e99aecb131574de8eee0591378
Oct 31 03:55:40.430958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.430934    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:55:40.637116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.637012    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerName="main" containerID="cri-o://fa48fdb2e316f4751a30a3287f787844385abe8fddaca1c2a435f5d767d5ee2e" gracePeriod=45
Oct 31 03:55:40.714280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.714238    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.244:8000/ready\": dial tcp 10.132.0.244:8000: connect: connection refused"
Oct 31 03:55:40.852054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.852009    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" event={"ID":"40d382c5-1a49-46d8-93d1-2cacdae68564","Type":"ContainerStarted","Data":"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"}
Oct 31 03:55:40.852315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.852291    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" event={"ID":"40d382c5-1a49-46d8-93d1-2cacdae68564","Type":"ContainerStarted","Data":"b257709a151dd919422863eecac68fa34daab08b41691e70237be546e0fe9ac0"}
Oct 31 03:55:40.852837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.852781    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:40.855163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.855134    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" event={"ID":"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a","Type":"ContainerStarted","Data":"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"}
Oct 31 03:55:40.855274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.855185    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" event={"ID":"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a","Type":"ContainerStarted","Data":"0ee73b7775c18f11f2e96961665618ce32f657e99aecb131574de8eee0591378"}
Oct 31 03:55:40.855274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.855213    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:40.857495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.857466    5098 generic.go:356] "Generic (PLEG): container finished" podID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerID="fa48fdb2e316f4751a30a3287f787844385abe8fddaca1c2a435f5d767d5ee2e" exitCode=0
Oct 31 03:55:40.857660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.857506    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" event={"ID":"674ae73c-9935-4daf-8f92-02209cd83fd7","Type":"ContainerDied","Data":"fa48fdb2e316f4751a30a3287f787844385abe8fddaca1c2a435f5d767d5ee2e"}
Oct 31 03:55:40.920361 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.920314    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" podStartSLOduration=1.920298704 podStartE2EDuration="1.920298704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:40.878617387 +0000 UTC m=+2415.403836970" watchObservedRunningTime="2026-10-31 03:55:40.920298704 +0000 UTC m=+2415.445518293"
Oct 31 03:55:40.920556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.920535    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" podStartSLOduration=1.920529698 podStartE2EDuration="1.920529698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:40.915736346 +0000 UTC m=+2415.440955934" watchObservedRunningTime="2026-10-31 03:55:40.920529698 +0000 UTC m=+2415.445749287"
Oct 31 03:55:40.957131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:40.957098    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:55:41.025498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025453    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.025498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025512    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.025942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025651    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.025942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025698    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\" (UniqueName: \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.025942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025723    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.025942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.025859    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\" (UniqueName: \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\") pod \"674ae73c-9935-4daf-8f92-02209cd83fd7\" (UID: \"674ae73c-9935-4daf-8f92-02209cd83fd7\") "
Oct 31 03:55:41.027254 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.027226    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:41.041716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.041669    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:41.042205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.042170    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:41.104133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.104004    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:41.104133 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.104105    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:41.105782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.105750    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf" pod "674ae73c-9935-4daf-8f92-02209cd83fd7" (UID: "674ae73c-9935-4daf-8f92-02209cd83fd7"). InnerVolumeSpecName "kube-api-access-tzjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:41.128431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128397    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/674ae73c-9935-4daf-8f92-02209cd83fd7-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.128431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128430    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.128709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128448    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.128709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128461    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.128709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128474    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzjnf\" (UniqueName: \"kubernetes.io/projected/674ae73c-9935-4daf-8f92-02209cd83fd7-kube-api-access-tzjnf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.128709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.128487    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/674ae73c-9935-4daf-8f92-02209cd83fd7-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:41.867942 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.867911    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"
Oct 31 03:55:41.868422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.867940    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd" event={"ID":"674ae73c-9935-4daf-8f92-02209cd83fd7","Type":"ContainerDied","Data":"59dbcab7a0cdcc203b1456270e09b45e3315c8c11ba8a79441cb570a0f7527d2"}
Oct 31 03:55:41.868422 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.867997    5098 scope.go:122] "RemoveContainer" containerID="fa48fdb2e316f4751a30a3287f787844385abe8fddaca1c2a435f5d767d5ee2e"
Oct 31 03:55:41.901994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.901946    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:55:41.902965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:41.902935    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-d4145b/auth-test-model-d4145b-kserve-6969f59f65-t4xkd"]
Oct 31 03:55:42.364682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.364597    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerName="payload-processing" containerID="cri-o://642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0" gracePeriod=25
Oct 31 03:55:42.381974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.381911    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" podUID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" containerName="payload-pre-processing" containerID="cri-o://3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7" gracePeriod=25
Oct 31 03:55:42.688376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.688343    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:55:42.760766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.760285    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:55:42.775016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.772030    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\" (UniqueName: \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\") pod \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") "
Oct 31 03:55:42.775016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.772222    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\") pod \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\" (UID: \"d442e0ea-dd06-4492-bf3b-6d65627ecd87\") "
Oct 31 03:55:42.775016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.773382    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume" pod "d442e0ea-dd06-4492-bf3b-6d65627ecd87" (UID: "d442e0ea-dd06-4492-bf3b-6d65627ecd87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:42.778813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.778780    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j" pod "d442e0ea-dd06-4492-bf3b-6d65627ecd87" (UID: "d442e0ea-dd06-4492-bf3b-6d65627ecd87"). InnerVolumeSpecName "kube-api-access-g9d7j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:42.833474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.833444    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:55:42.874716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.873600    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\" (UniqueName: \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\") pod \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") "
Oct 31 03:55:42.874716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.873720    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\") pod \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\" (UID: \"0d0f2052-ee90-4e30-9c88-81227ca3a5b8\") "
Oct 31 03:55:42.874716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.874593    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9d7j\" (UniqueName: \"kubernetes.io/projected/d442e0ea-dd06-4492-bf3b-6d65627ecd87-kube-api-access-g9d7j\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:42.874716 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.874636    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d442e0ea-dd06-4492-bf3b-6d65627ecd87-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:42.875455 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.875385    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume" pod "0d0f2052-ee90-4e30-9c88-81227ca3a5b8" (UID: "0d0f2052-ee90-4e30-9c88-81227ca3a5b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:42.880244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.880149    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj" pod "0d0f2052-ee90-4e30-9c88-81227ca3a5b8" (UID: "0d0f2052-ee90-4e30-9c88-81227ca3a5b8"). InnerVolumeSpecName "kube-api-access-m8fkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:42.881684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.881652    5098 generic.go:356] "Generic (PLEG): container finished" podID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerID="642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0" exitCode=0
Oct 31 03:55:42.881824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.881779    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" event={"ID":"d442e0ea-dd06-4492-bf3b-6d65627ecd87","Type":"ContainerDied","Data":"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"}
Oct 31 03:55:42.881901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.881832    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7" event={"ID":"d442e0ea-dd06-4492-bf3b-6d65627ecd87","Type":"ContainerDied","Data":"2e44c90e4357d0cfe32b28d1ca52603f6a1551fd75c8a77f2c83722c78883f54"}
Oct 31 03:55:42.881901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.881862    5098 scope.go:122] "RemoveContainer" containerID="642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"
Oct 31 03:55:42.882019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.881992    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"
Oct 31 03:55:42.889203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.889169    5098 generic.go:356] "Generic (PLEG): container finished" podID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerID="6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47" exitCode=0
Oct 31 03:55:42.889321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.889307    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" event={"ID":"8b021c19-9ed3-4036-956f-5747a64b5e93","Type":"ContainerDied","Data":"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"}
Oct 31 03:55:42.889402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.889356    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md" event={"ID":"8b021c19-9ed3-4036-956f-5747a64b5e93","Type":"ContainerDied","Data":"c9d39596925098b7409076c3af381adcc070c5c090d8aff80554f837b72d544b"}
Oct 31 03:55:42.889473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.889428    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"
Oct 31 03:55:42.893310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.893281    5098 generic.go:356] "Generic (PLEG): container finished" podID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerID="e5ab75a1bbc975eb9af8d6b6653df9cd9e4effe94643540f09ebffde9e20716d" exitCode=0
Oct 31 03:55:42.893691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.893534    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" event={"ID":"9262e9b7-5033-4443-894d-ebbc0bab390e","Type":"ContainerDied","Data":"e5ab75a1bbc975eb9af8d6b6653df9cd9e4effe94643540f09ebffde9e20716d"}
Oct 31 03:55:42.899465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.899431    5098 generic.go:356] "Generic (PLEG): container finished" podID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" containerID="3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7" exitCode=0
Oct 31 03:55:42.899636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.899588    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"
Oct 31 03:55:42.899726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.899661    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" event={"ID":"0d0f2052-ee90-4e30-9c88-81227ca3a5b8","Type":"ContainerDied","Data":"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"}
Oct 31 03:55:42.899789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.899736    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6" event={"ID":"0d0f2052-ee90-4e30-9c88-81227ca3a5b8","Type":"ContainerDied","Data":"e6d8d038fce0748d48d32a851e43dc39b9daaeafab566baf10ccaf4682dfb687"}
Oct 31 03:55:42.912705 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.912576    5098 scope.go:122] "RemoveContainer" containerID="642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"
Oct 31 03:55:42.913256 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:42.913003    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0\": container with ID starting with 642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0 not found: ID does not exist" containerID="642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"
Oct 31 03:55:42.913256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.913053    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0"} err="failed to get container status \"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0\": rpc error: code = NotFound desc = could not find container \"642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0\": container with ID starting with 642615fc3a28df49cc1818a5577779db76887b0ff193ca4b0a2beadaa64f0ad0 not found: ID does not exist"
Oct 31 03:55:42.913256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.913079    5098 scope.go:122] "RemoveContainer" containerID="6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"
Oct 31 03:55:42.920020 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.919989    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:55:42.925851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.925822    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-50b5f2-846db4d797-r7gj7"]
Oct 31 03:55:42.969820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.969736    5098 scope.go:122] "RemoveContainer" containerID="6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"
Oct 31 03:55:42.970112 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:42.970092    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47\": container with ID starting with 6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47 not found: ID does not exist" containerID="6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"
Oct 31 03:55:42.970190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.970127    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47"} err="failed to get container status \"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47\": rpc error: code = NotFound desc = could not find container \"6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47\": container with ID starting with 6f1f12741ec47ea67c469358a109315db9e8168bef4f1191c8056a7259b9cf47 not found: ID does not exist"
Oct 31 03:55:42.970190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.970155    5098 scope.go:122] "RemoveContainer" containerID="3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"
Oct 31 03:55:42.973790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.973764    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:55:42.975920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.975895    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\") pod \"8b021c19-9ed3-4036-956f-5747a64b5e93\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") "
Oct 31 03:55:42.976050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.975933    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\" (UniqueName: \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\") pod \"8b021c19-9ed3-4036-956f-5747a64b5e93\" (UID: \"8b021c19-9ed3-4036-956f-5747a64b5e93\") "
Oct 31 03:55:42.976267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.976246    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume" pod "8b021c19-9ed3-4036-956f-5747a64b5e93" (UID: "8b021c19-9ed3-4036-956f-5747a64b5e93"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:42.977279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.977253    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:42.977279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.977281    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b021c19-9ed3-4036-956f-5747a64b5e93-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:42.977469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.977291    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8fkj\" (UniqueName: \"kubernetes.io/projected/0d0f2052-ee90-4e30-9c88-81227ca3a5b8-kube-api-access-m8fkj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:42.978192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.978174    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2" pod "8b021c19-9ed3-4036-956f-5747a64b5e93" (UID: "8b021c19-9ed3-4036-956f-5747a64b5e93"). InnerVolumeSpecName "kube-api-access-x8dk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:42.984457 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.984427    5098 scope.go:122] "RemoveContainer" containerID="3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"
Oct 31 03:55:42.985686 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:42.985659    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7\": container with ID starting with 3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7 not found: ID does not exist" containerID="3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"
Oct 31 03:55:42.985796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.985693    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7"} err="failed to get container status \"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7\": rpc error: code = NotFound desc = could not find container \"3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7\": container with ID starting with 3745a2e07a449eb1c6e4eda5c317b5abd0bcaa109e9fae6d801ed99c492d58d7 not found: ID does not exist"
Oct 31 03:55:42.991569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.991543    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:55:42.996773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:42.996751    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-50b5f2-5f9476b8bc-ssxg6"]
Oct 31 03:55:43.078773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.078731    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\" (UniqueName: \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\") pod \"9262e9b7-5033-4443-894d-ebbc0bab390e\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") "
Oct 31 03:55:43.078982 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.078819    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\") pod \"9262e9b7-5033-4443-894d-ebbc0bab390e\" (UID: \"9262e9b7-5033-4443-894d-ebbc0bab390e\") "
Oct 31 03:55:43.079217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.079188    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume" pod "9262e9b7-5033-4443-894d-ebbc0bab390e" (UID: "9262e9b7-5033-4443-894d-ebbc0bab390e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:43.079681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.079661    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9262e9b7-5033-4443-894d-ebbc0bab390e-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:43.079768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.079689    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8dk2\" (UniqueName: \"kubernetes.io/projected/8b021c19-9ed3-4036-956f-5747a64b5e93-kube-api-access-x8dk2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:43.081203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.081174    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9" pod "9262e9b7-5033-4443-894d-ebbc0bab390e" (UID: "9262e9b7-5033-4443-894d-ebbc0bab390e"). InnerVolumeSpecName "kube-api-access-kpkd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:43.181504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.181415    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpkd9\" (UniqueName: \"kubernetes.io/projected/9262e9b7-5033-4443-894d-ebbc0bab390e-kube-api-access-kpkd9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:43.218501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.218466    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:55:43.223109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.223079    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5d9b6-5f7cb55bdc-bx4md"]
Oct 31 03:55:43.614142 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.614109    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" path="/var/lib/kubelet/pods/0d0f2052-ee90-4e30-9c88-81227ca3a5b8/volumes"
Oct 31 03:55:43.614669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.614652    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" path="/var/lib/kubelet/pods/674ae73c-9935-4daf-8f92-02209cd83fd7/volumes"
Oct 31 03:55:43.615112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.615095    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" path="/var/lib/kubelet/pods/8b021c19-9ed3-4036-956f-5747a64b5e93/volumes"
Oct 31 03:55:43.615910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.615887    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" path="/var/lib/kubelet/pods/d442e0ea-dd06-4492-bf3b-6d65627ecd87/volumes"
Oct 31 03:55:43.913076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.912990    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"
Oct 31 03:55:43.913076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.913021    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8" event={"ID":"9262e9b7-5033-4443-894d-ebbc0bab390e","Type":"ContainerDied","Data":"2981fed050c350623e7ad8074a7ab3de1c0ec79aa63b19c2008feecfd29e4ca8"}
Oct 31 03:55:43.913593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.913083    5098 scope.go:122] "RemoveContainer" containerID="e5ab75a1bbc975eb9af8d6b6653df9cd9e4effe94643540f09ebffde9e20716d"
Oct 31 03:55:43.939768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.939731    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:55:43.944155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:43.944128    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5d9b6-6d5686b7ff-82cb8"]
Oct 31 03:55:44.736878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:44.736821    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"]
Oct 31 03:55:44.741977 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:44.741933    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b81d65-q9k4s"]
Oct 31 03:55:45.224219 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.224177    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"]
Oct 31 03:55:45.228703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.228673    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-50b5f2-28rmn"]
Oct 31 03:55:45.614305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.614273    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25cdffa1-b007-4a78-b2eb-baf53fd39cfb" path="/var/lib/kubelet/pods/25cdffa1-b007-4a78-b2eb-baf53fd39cfb/volumes"
Oct 31 03:55:45.614812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.614797    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50312768-d894-4d93-a05b-2bb3c89b907c" path="/var/lib/kubelet/pods/50312768-d894-4d93-a05b-2bb3c89b907c/volumes"
Oct 31 03:55:45.615241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.615226    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" path="/var/lib/kubelet/pods/9262e9b7-5033-4443-894d-ebbc0bab390e/volumes"
Oct 31 03:55:45.790656 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:45.790620    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:55:49.451666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.450312    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:55:49.550385 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.533033    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:55:49.564458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.564418    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy" containerID="cri-o://b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41" gracePeriod=30
Oct 31 03:55:49.707278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.707184    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:55:49.755360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.755319    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"]
Oct 31 03:55:49.758580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.758537    5098 cpu_manager.go:414] "containerMap: removing container" podUID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" containerName="payload-pre-processing"
Oct 31 03:55:49.758824 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.758804    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" containerName="payload-pre-processing"
Oct 31 03:55:49.758978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.758966    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing"
Oct 31 03:55:49.759054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759043    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing"
Oct 31 03:55:49.759193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759178    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerName="payload-processing"
Oct 31 03:55:49.759292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759280    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerName="payload-processing"
Oct 31 03:55:49.759408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759398    5098 cpu_manager.go:414] "containerMap: removing container" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerName="main"
Oct 31 03:55:49.759481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759472    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerName="main"
Oct 31 03:55:49.759649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759590    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing"
Oct 31 03:55:49.759769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.759755    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing"
Oct 31 03:55:49.761305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.761279    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d442e0ea-dd06-4492-bf3b-6d65627ecd87" containerName="payload-processing"
Oct 31 03:55:49.761642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.761627    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="674ae73c-9935-4daf-8f92-02209cd83fd7" containerName="main"
Oct 31 03:55:49.761791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.761776    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9262e9b7-5033-4443-894d-ebbc0bab390e" containerName="payload-pre-processing"
Oct 31 03:55:49.761941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.761928    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d0f2052-ee90-4e30-9c88-81227ca3a5b8" containerName="payload-pre-processing"
Oct 31 03:55:49.762044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.762034    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b021c19-9ed3-4036-956f-5747a64b5e93" containerName="payload-processing"
Oct 31 03:55:49.771185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.771150    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"]
Oct 31 03:55:49.771400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.771388    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"]
Oct 31 03:55:49.775353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.775327    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.788936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.779445    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"]
Oct 31 03:55:49.789941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.779532    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.796024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.795995    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:55:49.796305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.796290    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:49.863799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.863764    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.863991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.863829    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64lcq\" (UniqueName: \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.864043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.863978    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctp6\" (UniqueName: \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.864163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.864141    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.864324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.864304    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.864404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.864387    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.965978 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.965878    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.966160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.965988    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.966160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966033    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.966160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966083    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.966160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966122    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64lcq\" (UniqueName: \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.966405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966242    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctp6\" (UniqueName: \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.966802 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966768    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.967023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.966840    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.968646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.968621    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.968791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.968773    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:49.975312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.975282    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64lcq\" (UniqueName: \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:49.975418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:49.975396    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctp6\" (UniqueName: \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:50.119356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.119316    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:50.125837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.125810    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:50.418172 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:50.418139    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001a5098_8bf4_4522_9807_e9db182c5b9c.slice/crio-16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5 WatchSource:0}: Error finding container 16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5: Status 404 returned error can't find the container with id 16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5
Oct 31 03:55:50.418597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.418573    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"]
Oct 31 03:55:50.443426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.443391    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"]
Oct 31 03:55:50.740109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.740000    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw" event={"ID":"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c","Type":"ContainerStarted","Data":"1cf564b0a495750ddd1ae5081bd12649739f472532a8b7f27645ea622bb80a13"}
Oct 31 03:55:50.741688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:50.741645    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz" event={"ID":"001a5098-8bf4-4522-9807-e9db182c5b9c","Type":"ContainerStarted","Data":"16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5"}
Oct 31 03:55:51.752178 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.752087    5098 generic.go:356] "Generic (PLEG): container finished" podID="44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" containerID="8864a8caff0ed5c64dd33d72482f56234d0834608a70495715449cbab7ad2417" exitCode=0
Oct 31 03:55:51.752645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.752168    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw" event={"ID":"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c","Type":"ContainerDied","Data":"8864a8caff0ed5c64dd33d72482f56234d0834608a70495715449cbab7ad2417"}
Oct 31 03:55:51.754023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.753994    5098 generic.go:356] "Generic (PLEG): container finished" podID="001a5098-8bf4-4522-9807-e9db182c5b9c" containerID="88dcd74d8bff8570df443b09e8bb38781bf50f724b46798d07475dce8e37b1fa" exitCode=0
Oct 31 03:55:51.754122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.754078    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz" event={"ID":"001a5098-8bf4-4522-9807-e9db182c5b9c","Type":"ContainerDied","Data":"88dcd74d8bff8570df443b09e8bb38781bf50f724b46798d07475dce8e37b1fa"}
Oct 31 03:55:51.871338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.871306    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:55:51.871537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:51.871398    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:55:52.995559 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:52.995522    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:53.050140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.050112    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:53.115621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115562    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\" (UniqueName: \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\") pod \"001a5098-8bf4-4522-9807-e9db182c5b9c\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") "
Oct 31 03:55:53.115818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115775    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\") pod \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") "
Oct 31 03:55:53.115818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115813    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\" (UniqueName: \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\") pod \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") "
Oct 31 03:55:53.115961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115855    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\") pod \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\" (UID: \"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c\") "
Oct 31 03:55:53.115961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115898    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\") pod \"001a5098-8bf4-4522-9807-e9db182c5b9c\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") "
Oct 31 03:55:53.115961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.115944    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\") pod \"001a5098-8bf4-4522-9807-e9db182c5b9c\" (UID: \"001a5098-8bf4-4522-9807-e9db182c5b9c\") "
Oct 31 03:55:53.116206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.116181    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca" pod "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" (UID: "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:53.116411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.116383    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca" pod "001a5098-8bf4-4522-9807-e9db182c5b9c" (UID: "001a5098-8bf4-4522-9807-e9db182c5b9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:53.116887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.116858    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.116994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.116893    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.118296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.118268    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca" pod "001a5098-8bf4-4522-9807-e9db182c5b9c" (UID: "001a5098-8bf4-4522-9807-e9db182c5b9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:53.118532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.118511    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq" pod "001a5098-8bf4-4522-9807-e9db182c5b9c" (UID: "001a5098-8bf4-4522-9807-e9db182c5b9c"). InnerVolumeSpecName "kube-api-access-64lcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:53.118653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.118542    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6" pod "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" (UID: "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c"). InnerVolumeSpecName "kube-api-access-tctp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:53.118941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.118925    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca" pod "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" (UID: "44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:55:53.218366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.218333    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctp6\" (UniqueName: \"kubernetes.io/projected/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-kube-api-access-tctp6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.218366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.218363    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.218366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.218373    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/001a5098-8bf4-4522-9807-e9db182c5b9c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.218631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.218382    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64lcq\" (UniqueName: \"kubernetes.io/projected/001a5098-8bf4-4522-9807-e9db182c5b9c-kube-api-access-64lcq\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:53.837732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.835584    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw" event={"ID":"44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c","Type":"ContainerDied","Data":"1cf564b0a495750ddd1ae5081bd12649739f472532a8b7f27645ea622bb80a13"}
Oct 31 03:55:53.837732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.835681    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cf564b0a495750ddd1ae5081bd12649739f472532a8b7f27645ea622bb80a13"
Oct 31 03:55:53.837732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.835698    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"
Oct 31 03:55:53.844554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.844504    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz" event={"ID":"001a5098-8bf4-4522-9807-e9db182c5b9c","Type":"ContainerDied","Data":"16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5"}
Oct 31 03:55:53.844778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.844579    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16fb4a605012a20b0697a1e1cf57cc3936ab6b9dc940a10af76c270a599924f5"
Oct 31 03:55:53.844778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:53.844675    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"
Oct 31 03:55:54.824988 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:55:54.824952    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf274d_d1ed_4778_b674_cc6ad7e4b82e.slice/crio-b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ecf274d_d1ed_4778_b674_cc6ad7e4b82e.slice/crio-conmon-b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41.scope\": RecentStats: unable to find data in memory cache]"
Oct 31 03:55:54.860372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:54.860304    5098 generic.go:356] "Generic (PLEG): container finished" podID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerID="b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41" exitCode=0
Oct 31 03:55:54.860941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:54.860818    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" event={"ID":"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e","Type":"ContainerDied","Data":"b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41"}
Oct 31 03:55:55.081405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.081373    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:55:55.175668 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.175044    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.177449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.176097    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.177886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.177402    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:55.178670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178572    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.178795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178742    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.178795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178778    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.179009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178815    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.179009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178868    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.179009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178922    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.179009 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.178954    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\") pod \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\" (UID: \"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e\") "
Oct 31 03:55:55.179284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179238    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:55.179731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179703    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:55.179848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179732    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:55.179963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179943    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.180048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179967    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.180048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179981    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.180048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.179995    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.180575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.180160    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:55:55.188145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.188115    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:55:55.188267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.188122    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:55.189106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.189069    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "kube-api-access-9j6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:55:55.189753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.189725    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy" pod "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" (UID: "7ecf274d-d1ed-4778-b674-cc6ad7e4b82e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:55:55.281632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.281564    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.281632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.281601    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j6kv\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-kube-api-access-9j6kv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.281862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.281647    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.281862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.281656    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.281862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.281664    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:55:55.789774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.789732    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:55:55.791657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791602    5098 cpu_manager.go:414] "containerMap: removing container" podUID="44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" containerName="revoke-keys"
Oct 31 03:55:55.791657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791657    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" containerName="revoke-keys"
Oct 31 03:55:55.791866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791688    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy"
Oct 31 03:55:55.791866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791697    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy"
Oct 31 03:55:55.791866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791726    5098 cpu_manager.go:414] "containerMap: removing container" podUID="001a5098-8bf4-4522-9807-e9db182c5b9c" containerName="revoke-keys"
Oct 31 03:55:55.791866 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791734    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="001a5098-8bf4-4522-9807-e9db182c5b9c" containerName="revoke-keys"
Oct 31 03:55:55.792072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791927    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="001a5098-8bf4-4522-9807-e9db182c5b9c" containerName="revoke-keys"
Oct 31 03:55:55.792072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791954    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" containerName="revoke-keys"
Oct 31 03:55:55.792072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.791967    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" containerName="istio-proxy"
Oct 31 03:55:55.807462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.803003    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:55:55.807462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.803138    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:55.809667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.808583    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-39e8cf-serving-cert\""
Oct 31 03:55:55.872566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.872476    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"
Oct 31 03:55:55.873047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.872472    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8" event={"ID":"7ecf274d-d1ed-4778-b674-cc6ad7e4b82e","Type":"ContainerDied","Data":"d43298cd0ed4357932b6eca7a178f9d22e8bd4bbe2dfd49b4ed6023b2badee8d"}
Oct 31 03:55:55.873047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.872676    5098 scope.go:122] "RemoveContainer" containerID="b5e03e0dd7d6790f66e7027eb6d0e63bc27a5d7400e4b975204672d046abaf41"
Oct 31 03:55:55.896220 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.896192    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:55:55.900816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:55.900786    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-d4145b-openshift-default-5b64679d7-kqwj8"]
Oct 31 03:55:56.010629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.008962    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.010629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.009092    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.010629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.009254    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.010629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.009326    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26sg7\" (UniqueName: \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.110901 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.110867    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.111068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.110917    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26sg7\" (UniqueName: \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.111068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.111061    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.111218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.111197    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.111655 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.111586    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.113813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.113784    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.113958 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.113936    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.119731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.119706    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26sg7\" (UniqueName: \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\") pod \"maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.130683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.130600    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:56.564628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.561491    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:55:56.579661 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:56.576262    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91926258_2c8b_4918_980e_d066b6b68843.slice/crio-2d56b17e0688274ed549eb42b28a8c365909415d69f8b272f72c5ebed711d404 WatchSource:0}: Error finding container 2d56b17e0688274ed549eb42b28a8c365909415d69f8b272f72c5ebed711d404: Status 404 returned error can't find the container with id 2d56b17e0688274ed549eb42b28a8c365909415d69f8b272f72c5ebed711d404
Oct 31 03:55:56.602651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.594080    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 03:55:56.884007 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:56.883919    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" event={"ID":"91926258-2c8b-4918-980e-d066b6b68843","Type":"ContainerStarted","Data":"2d56b17e0688274ed549eb42b28a8c365909415d69f8b272f72c5ebed711d404"}
Oct 31 03:55:57.278810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.278774    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:55:57.279160 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.279131    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy" containerID="cri-o://580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55" gracePeriod=30
Oct 31 03:55:57.619537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.619501    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ecf274d-d1ed-4778-b674-cc6ad7e4b82e" path="/var/lib/kubelet/pods/7ecf274d-d1ed-4778-b674-cc6ad7e4b82e/volumes"
Oct 31 03:55:57.808632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.803009    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"]
Oct 31 03:55:57.825702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.823244    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.828016 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.827979    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"]
Oct 31 03:55:57.831778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.831751    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:55:57.832068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.832049    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:55:57.895862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.895780    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn872\" (UniqueName: \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.896348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.896066    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.896348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.896124    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.896508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.896475    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" event={"ID":"91926258-2c8b-4918-980e-d066b6b68843","Type":"ContainerStarted","Data":"fc7aadb742efbca38406d79eac2730e506ecb5a68487885fa68dc9eefd2fb53c"}
Oct 31 03:55:57.896587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.896571    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:55:57.915511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.915459    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" podStartSLOduration=2.304355142 podStartE2EDuration="2.91544485s" totalImagesPullingTime="611.089708ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:57.913421144 +0000 UTC m=+2432.438640732" watchObservedRunningTime="2026-10-31 03:55:57.91544485 +0000 UTC m=+2432.440664435"
Oct 31 03:55:57.997364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.997332    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn872\" (UniqueName: \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.997590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.997573    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.997729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.997660    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:57.998368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:57.998335    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:58.000949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.000924    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:58.009564 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.009536    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn872\" (UniqueName: \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\") pod \"maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:58.151725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.149114    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:55:58.594848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.594818    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"]
Oct 31 03:55:58.598314 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:55:58.598285    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb100944e_7e6f_4e9f_8027_215801affb6c.slice/crio-71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd WatchSource:0}: Error finding container 71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd: Status 404 returned error can't find the container with id 71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd
Oct 31 03:55:58.601379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.601352    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.602026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.601999    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"]
Oct 31 03:55:58.609273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.609240    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"]
Oct 31 03:55:58.631151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.631118    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.631601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.631568    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.631778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.631743    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57vss\" (UniqueName: \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.736479 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.736429    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.736713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.736591    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57vss\" (UniqueName: \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.736915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.736859    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.737973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.737810    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.741366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.741340    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.757806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.757778    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57vss\" (UniqueName: \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\") pod \"maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:58.849700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.839857    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:55:58.849700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.846231    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.862175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.862123    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-20585c-openshift-default-dockercfg-mnxkq\""
Oct 31 03:55:58.863398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.863368    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:55:58.869913 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.869887    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.871165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.870957    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:55:58.897402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.897369    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:55:58.914240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.914207    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs" event={"ID":"b100944e-7e6f-4e9f-8027-215801affb6c","Type":"ContainerStarted","Data":"71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd"}
Oct 31 03:55:58.941496 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941450    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.941708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941523    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.941783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941702    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.941880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941857    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.941938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941893    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.941938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.941927    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942044    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdk2\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.942168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942147    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942187    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.942297 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942276    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942326    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.942417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942389    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942465    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.942533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942494    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942531    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942556    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48gp\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942679    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:58.942790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.942735    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:58.974364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:58.974324    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044647    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044709    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044739    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044796    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdk2\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044874    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044908    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.044981    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045028    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045058    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045072 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045062    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045166    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045202    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045261    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045299    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48gp\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045347    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045403    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045462    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045466    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045577    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045718    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.045761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045742    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.046357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.045990    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.046357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.046261    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.046475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.046437    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.046535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.046497    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.047106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.047079    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.048587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.047756    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.048587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.048326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.049256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.049211    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.051967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.051942    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.052106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.051942    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.054081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.054056    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.054779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.054627    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.056231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.055551    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdk2\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.056231 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.055962    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.056676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.056592    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48gp\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\") pod \"e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") " pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.213314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.213272    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:55:59.216689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.216602    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:55:59.271271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.269965    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"]
Oct 31 03:55:59.575049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.573693    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:55:59.577585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.577452    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:55:59.580421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.580386    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.580511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.580463    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.580511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.580490    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.581368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.580998    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.581368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.581053    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.581368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.581083    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:55:59.673370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.673345    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:55:59.688255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.688219    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:55:59.928045 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.928000    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" event={"ID":"164f671b-b44e-4893-8b64-51baf403ba2e","Type":"ContainerStarted","Data":"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"}
Oct 31 03:55:59.928567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.928077    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" event={"ID":"164f671b-b44e-4893-8b64-51baf403ba2e","Type":"ContainerStarted","Data":"f32667a0a7e2508dac98df72881e758e45f416f182adfc0fd04edf4533c95f33"}
Oct 31 03:55:59.932878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.932004    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" event={"ID":"11027739-4566-428c-b85b-c84ae5513314","Type":"ContainerStarted","Data":"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"}
Oct 31 03:55:59.932878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.932067    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" event={"ID":"11027739-4566-428c-b85b-c84ae5513314","Type":"ContainerStarted","Data":"6711933cb91e12d6786412128ab3e7850eb8e0e514116d7bd3f8501d1483617f"}
Oct 31 03:55:59.935436 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.935400    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg" event={"ID":"373fdb48-6cdb-4e1e-8f78-37939255caac","Type":"ContainerStarted","Data":"c9f02c142e8fb83f3d8f1e524cba84ef83a1a31a6220a3d61cc73be8ad8c8ace"}
Oct 31 03:55:59.935677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.935467    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg" event={"ID":"373fdb48-6cdb-4e1e-8f78-37939255caac","Type":"ContainerStarted","Data":"1154f478e1f1311b2ccd0dd66343e1a86db1bc37cde40528601155d1acf00507"}
Oct 31 03:55:59.937949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.937916    5098 generic.go:356] "Generic (PLEG): container finished" podID="b100944e-7e6f-4e9f-8027-215801affb6c" containerID="63addc9cb88dfe52abce653c8b8bc2b37fc9f8ea856bca16905bb89fab3cde4a" exitCode=0
Oct 31 03:55:59.938181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.937956    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs" event={"ID":"b100944e-7e6f-4e9f-8027-215801affb6c","Type":"ContainerDied","Data":"63addc9cb88dfe52abce653c8b8bc2b37fc9f8ea856bca16905bb89fab3cde4a"}
Oct 31 03:55:59.956139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.956068    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podStartSLOduration=1.956046524 podStartE2EDuration="1.956046524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:55:59.951651902 +0000 UTC m=+2434.476871498" watchObservedRunningTime="2026-10-31 03:55:59.956046524 +0000 UTC m=+2434.481266115"
Oct 31 03:55:59.979634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:55:59.969255    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:56:00.239697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.118884    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg" podStartSLOduration=2.118863639 podStartE2EDuration="2.118863639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:00.003921402 +0000 UTC m=+2434.529140992" watchObservedRunningTime="2026-10-31 03:56:00.118863639 +0000 UTC m=+2434.644083231"
Oct 31 03:56:00.300919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.234172    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:56:00.300919 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.278841    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:56:00.325628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.324003    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:00.325628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.324068    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:00.325628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.324446    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:00.325628 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.324482    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:00.362774 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:00.362703    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podStartSLOduration=2.362679327 podStartE2EDuration="2.362679327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:00.05840897 +0000 UTC m=+2434.583628560" watchObservedRunningTime="2026-10-31 03:56:00.362679327 +0000 UTC m=+2434.887898916"
Oct 31 03:56:01.035998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.035951    5098 generic.go:356] "Generic (PLEG): container finished" podID="373fdb48-6cdb-4e1e-8f78-37939255caac" containerID="c9f02c142e8fb83f3d8f1e524cba84ef83a1a31a6220a3d61cc73be8ad8c8ace" exitCode=0
Oct 31 03:56:01.036575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.036533    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg" event={"ID":"373fdb48-6cdb-4e1e-8f78-37939255caac","Type":"ContainerDied","Data":"c9f02c142e8fb83f3d8f1e524cba84ef83a1a31a6220a3d61cc73be8ad8c8ace"}
Oct 31 03:56:01.216653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.213764    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:01.218810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.218776    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:01.219041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.219011    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:01.239863 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:01.235839    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:56:01.318962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.317143    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:01.615525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.615494    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:56:01.643490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.643454    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\") pod \"b100944e-7e6f-4e9f-8027-215801affb6c\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") "
Oct 31 03:56:01.643704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.643643    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\") pod \"b100944e-7e6f-4e9f-8027-215801affb6c\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") "
Oct 31 03:56:01.643799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.643761    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\" (UniqueName: \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\") pod \"b100944e-7e6f-4e9f-8027-215801affb6c\" (UID: \"b100944e-7e6f-4e9f-8027-215801affb6c\") "
Oct 31 03:56:01.644260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.644233    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca" pod "b100944e-7e6f-4e9f-8027-215801affb6c" (UID: "b100944e-7e6f-4e9f-8027-215801affb6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:01.644543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.644523    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:01.646688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.646660    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca" pod "b100944e-7e6f-4e9f-8027-215801affb6c" (UID: "b100944e-7e6f-4e9f-8027-215801affb6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:01.646794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.646749    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872" pod "b100944e-7e6f-4e9f-8027-215801affb6c" (UID: "b100944e-7e6f-4e9f-8027-215801affb6c"). InnerVolumeSpecName "kube-api-access-mn872". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:01.745719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.745678    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn872\" (UniqueName: \"kubernetes.io/projected/b100944e-7e6f-4e9f-8027-215801affb6c-kube-api-access-mn872\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:01.745719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:01.745715    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b100944e-7e6f-4e9f-8027-215801affb6c-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:02.047980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.047947    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"
Oct 31 03:56:02.047980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.047963    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs" event={"ID":"b100944e-7e6f-4e9f-8027-215801affb6c","Type":"ContainerDied","Data":"71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd"}
Oct 31 03:56:02.048535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.048040    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71b9946873ab05ee21ec091058b9086cf8de9876a5a29753690ce633b1d8fdcd"
Oct 31 03:56:02.215983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.213920    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:02.215983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.214012    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:02.220941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.220730    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:02.220941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.220803    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:02.411027 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.410995    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:56:02.512551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.512499    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\" (UniqueName: \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\") pod \"373fdb48-6cdb-4e1e-8f78-37939255caac\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") "
Oct 31 03:56:02.513520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.513486    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\") pod \"373fdb48-6cdb-4e1e-8f78-37939255caac\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") "
Oct 31 03:56:02.514117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.514098    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\") pod \"373fdb48-6cdb-4e1e-8f78-37939255caac\" (UID: \"373fdb48-6cdb-4e1e-8f78-37939255caac\") "
Oct 31 03:56:02.514776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.514743    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca" pod "373fdb48-6cdb-4e1e-8f78-37939255caac" (UID: "373fdb48-6cdb-4e1e-8f78-37939255caac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:02.516026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.515998    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:02.517842 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.517803    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss" pod "373fdb48-6cdb-4e1e-8f78-37939255caac" (UID: "373fdb48-6cdb-4e1e-8f78-37939255caac"). InnerVolumeSpecName "kube-api-access-57vss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:02.520117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.520083    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca" pod "373fdb48-6cdb-4e1e-8f78-37939255caac" (UID: "373fdb48-6cdb-4e1e-8f78-37939255caac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:02.625207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.624665    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57vss\" (UniqueName: \"kubernetes.io/projected/373fdb48-6cdb-4e1e-8f78-37939255caac-kube-api-access-57vss\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:02.625207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.624700    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/373fdb48-6cdb-4e1e-8f78-37939255caac-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:02.922322 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.922291    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965140    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965300    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965338    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965445    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965625    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965665    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965700    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:02.980631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:02.965748    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\") pod \"252d74ea-9921-4058-8ca6-1d27167967fa\" (UID: \"252d74ea-9921-4058-8ca6-1d27167967fa\") "
Oct 31 03:56:03.012659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.011776    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:03.016397 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.014646    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.017126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.016650    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:03.017126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.016956    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:03.017535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.017364    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:03.017535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.017386    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:03.017841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.017598    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:03.020954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.020923    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:03.029490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.029459    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:03.029889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.029839    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj" pod "252d74ea-9921-4058-8ca6-1d27167967fa" (UID: "252d74ea-9921-4058-8ca6-1d27167967fa"). InnerVolumeSpecName "kube-api-access-cgtgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:03.112033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.111986    5098 generic.go:356] "Generic (PLEG): container finished" podID="252d74ea-9921-4058-8ca6-1d27167967fa" containerID="580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55" exitCode=0
Oct 31 03:56:03.112418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.112174    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"
Oct 31 03:56:03.112554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.112517    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" event={"ID":"252d74ea-9921-4058-8ca6-1d27167967fa","Type":"ContainerDied","Data":"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"}
Oct 31 03:56:03.112644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.112576    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq" event={"ID":"252d74ea-9921-4058-8ca6-1d27167967fa","Type":"ContainerDied","Data":"ebddd2094153d03e1a43955f9c8f70d4763416e407e5ba14c5d394275305e631"}
Oct 31 03:56:03.112644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.112616    5098 scope.go:122] "RemoveContainer" containerID="580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"
Oct 31 03:56:03.114869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.114807    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"
Oct 31 03:56:03.114967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.114830    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg" event={"ID":"373fdb48-6cdb-4e1e-8f78-37939255caac","Type":"ContainerDied","Data":"1154f478e1f1311b2ccd0dd66343e1a86db1bc37cde40528601155d1acf00507"}
Oct 31 03:56:03.115023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.114976    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1154f478e1f1311b2ccd0dd66343e1a86db1bc37cde40528601155d1acf00507"
Oct 31 03:56:03.116277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116251    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116405    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116514    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116597    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116711    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgtgj\" (UniqueName: \"kubernetes.io/projected/252d74ea-9921-4058-8ca6-1d27167967fa-kube-api-access-cgtgj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116733    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/252d74ea-9921-4058-8ca6-1d27167967fa-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116747    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/252d74ea-9921-4058-8ca6-1d27167967fa-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.116909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.116761    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/252d74ea-9921-4058-8ca6-1d27167967fa-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:03.129512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.129489    5098 scope.go:122] "RemoveContainer" containerID="580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"
Oct 31 03:56:03.129946 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:03.129922    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55\": container with ID starting with 580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55 not found: ID does not exist" containerID="580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"
Oct 31 03:56:03.130043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.129957    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55"} err="failed to get container status \"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55\": rpc error: code = NotFound desc = could not find container \"580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55\": container with ID starting with 580f0c900b5f53fd3a7021d10aa1aad522d0ce680d3096ac5e9419ea58fc7d55 not found: ID does not exist"
Oct 31 03:56:03.205784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.204659    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:56:03.209703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.209647    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b81d65-openshift-default-9d946fc79-6b8lq"]
Oct 31 03:56:03.214136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.214105    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:03.214305 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.214163    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:03.217551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.217519    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:03.217682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.217574    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:03.652634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.649012    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" path="/var/lib/kubelet/pods/252d74ea-9921-4058-8ca6-1d27167967fa/volumes"
Oct 31 03:56:03.922899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:03.922816    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:56:04.251702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.231791    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:04.251702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.231857    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:04.251702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.232386    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:04.251702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.232436    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:04.685632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.678378    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" podUID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" containerName="payload-processing" containerID="cri-o://9ed2396eb4e98102635a9ea166154c5f18ed41db1b864a3bfa7d98f47aca9521" gracePeriod=25
Oct 31 03:56:04.696637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.694694    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" podUID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" containerName="payload-pre-processing" containerID="cri-o://d580e2387365df950472fc9b295e8d9a777a7ab50d26912115e6c3d38233cae3" gracePeriod=25
Oct 31 03:56:04.979701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:04.969704    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" podUID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" containerName="payload-processing" containerID="cri-o://97130b945d7adb42185eb5540d5024cbcfca57d29d485e24336d426811db80c4" gracePeriod=25
Oct 31 03:56:05.144589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.144333    5098 generic.go:356] "Generic (PLEG): container finished" podID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" containerID="d580e2387365df950472fc9b295e8d9a777a7ab50d26912115e6c3d38233cae3" exitCode=0
Oct 31 03:56:05.144589 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.144499    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" event={"ID":"528eb2c5-8091-4f0c-b8a5-cb98a68e255f","Type":"ContainerDied","Data":"d580e2387365df950472fc9b295e8d9a777a7ab50d26912115e6c3d38233cae3"}
Oct 31 03:56:05.148697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.148430    5098 generic.go:356] "Generic (PLEG): container finished" podID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" containerID="9ed2396eb4e98102635a9ea166154c5f18ed41db1b864a3bfa7d98f47aca9521" exitCode=0
Oct 31 03:56:05.148697 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.148531    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" event={"ID":"b1bfcaa4-9246-4137-b4d4-83ab590f40b8","Type":"ContainerDied","Data":"9ed2396eb4e98102635a9ea166154c5f18ed41db1b864a3bfa7d98f47aca9521"}
Oct 31 03:56:05.214999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.214027    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:05.214999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.214104    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:05.217679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.217336    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:05.217679 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.217385    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:05.320344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.320311    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:56:05.418647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.382505    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\" (UniqueName: \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\") pod \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") "
Oct 31 03:56:05.418647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.383008    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\") pod \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\" (UID: \"528eb2c5-8091-4f0c-b8a5-cb98a68e255f\") "
Oct 31 03:56:05.419701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.419366    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume" pod "528eb2c5-8091-4f0c-b8a5-cb98a68e255f" (UID: "528eb2c5-8091-4f0c-b8a5-cb98a68e255f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:05.419701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.419379    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:56:05.420475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.420033    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" podUID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" containerName="maas-api" containerID="cri-o://60b4c6eb6512954fd98425d854723a838dce877c6f0cbd11fd0a5e619b2901d3" gracePeriod=30
Oct 31 03:56:05.421860 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.421816    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29" pod "528eb2c5-8091-4f0c-b8a5-cb98a68e255f" (UID: "528eb2c5-8091-4f0c-b8a5-cb98a68e255f"). InnerVolumeSpecName "kube-api-access-mjd29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:05.508432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.507652    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:05.513168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.512834    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjd29\" (UniqueName: \"kubernetes.io/projected/528eb2c5-8091-4f0c-b8a5-cb98a68e255f-kube-api-access-mjd29\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:05.520182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.519759    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:56:05.627198 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.627144    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\" (UniqueName: \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\") pod \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") "
Oct 31 03:56:05.632633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.627551    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\") pod \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\" (UID: \"b1bfcaa4-9246-4137-b4d4-83ab590f40b8\") "
Oct 31 03:56:05.633081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.633046    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume" pod "b1bfcaa4-9246-4137-b4d4-83ab590f40b8" (UID: "b1bfcaa4-9246-4137-b4d4-83ab590f40b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:05.642146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.642070    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn" pod "b1bfcaa4-9246-4137-b4d4-83ab590f40b8" (UID: "b1bfcaa4-9246-4137-b4d4-83ab590f40b8"). InnerVolumeSpecName "kube-api-access-hmmbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:05.804400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.735343    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmmbn\" (UniqueName: \"kubernetes.io/projected/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-kube-api-access-hmmbn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:05.804400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:05.735381    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1bfcaa4-9246-4137-b4d4-83ab590f40b8-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.112853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.112775    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"]
Oct 31 03:56:06.113083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.113066    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-63258b-pc7gz"]
Oct 31 03:56:06.265629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.264814    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused" start-of-body=
Oct 31 03:56:06.265629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.264894    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.20:15021/healthz/ready\": dial tcp 10.132.1.20:15021: connect: connection refused"
Oct 31 03:56:06.284538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.284489    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8" event={"ID":"528eb2c5-8091-4f0c-b8a5-cb98a68e255f","Type":"ContainerDied","Data":"1a37ceeec16027ec4eb5173835710177f84f654c5c7e605fb237aba205281c7a"}
Oct 31 03:56:06.285013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.284989    5098 scope.go:122] "RemoveContainer" containerID="d580e2387365df950472fc9b295e8d9a777a7ab50d26912115e6c3d38233cae3"
Oct 31 03:56:06.285440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.285417    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"
Oct 31 03:56:06.319951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.317900    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused" start-of-body=
Oct 31 03:56:06.319951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.317959    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.19:15021/healthz/ready\": dial tcp 10.132.1.19:15021: connect: connection refused"
Oct 31 03:56:06.323922 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.323759    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc" event={"ID":"b1bfcaa4-9246-4137-b4d4-83ab590f40b8","Type":"ContainerDied","Data":"830f6185203fb8b4f3e28cf71b96ac9d354c1873bd1df6c72e52e664e34ae90d"}
Oct 31 03:56:06.325084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.324713    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"
Oct 31 03:56:06.329237 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.329166    5098 generic.go:356] "Generic (PLEG): container finished" podID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" containerID="97130b945d7adb42185eb5540d5024cbcfca57d29d485e24336d426811db80c4" exitCode=0
Oct 31 03:56:06.329621 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.329268    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" event={"ID":"f6e8985f-f4fe-45bd-a46d-1dacb4865275","Type":"ContainerDied","Data":"97130b945d7adb42185eb5540d5024cbcfca57d29d485e24336d426811db80c4"}
Oct 31 03:56:06.329936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.329919    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2" event={"ID":"f6e8985f-f4fe-45bd-a46d-1dacb4865275","Type":"ContainerDied","Data":"adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014"}
Oct 31 03:56:06.330224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.330208    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adbc3578c1a2edb956b1f5fc7eb043eb7ac3f23e8add47710f859919f7803014"
Oct 31 03:56:06.331339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.331315    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:56:06.334058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.334033    5098 generic.go:356] "Generic (PLEG): container finished" podID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" containerID="60b4c6eb6512954fd98425d854723a838dce877c6f0cbd11fd0a5e619b2901d3" exitCode=0
Oct 31 03:56:06.334352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.334336    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" event={"ID":"07c8d566-0a7f-43b7-bb17-cfe678e2ebae","Type":"ContainerDied","Data":"60b4c6eb6512954fd98425d854723a838dce877c6f0cbd11fd0a5e619b2901d3"}
Oct 31 03:56:06.402694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.402658    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:56:06.407644 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.407598    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-63258b-789c8545ff-sdvx8"]
Oct 31 03:56:06.408174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.408011    5098 scope.go:122] "RemoveContainer" containerID="9ed2396eb4e98102635a9ea166154c5f18ed41db1b864a3bfa7d98f47aca9521"
Oct 31 03:56:06.441999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.441968    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\") pod \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") "
Oct 31 03:56:06.442409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.442383    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\" (UniqueName: \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\") pod \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\" (UID: \"f6e8985f-f4fe-45bd-a46d-1dacb4865275\") "
Oct 31 03:56:06.442868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.442484    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume" pod "f6e8985f-f4fe-45bd-a46d-1dacb4865275" (UID: "f6e8985f-f4fe-45bd-a46d-1dacb4865275"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:06.443761 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.443404    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6e8985f-f4fe-45bd-a46d-1dacb4865275-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.446261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.446234    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp" pod "f6e8985f-f4fe-45bd-a46d-1dacb4865275" (UID: "f6e8985f-f4fe-45bd-a46d-1dacb4865275"). InnerVolumeSpecName "kube-api-access-ns2cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:06.532641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.532160    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:56:06.539636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.539585    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-63258b-859b5f6c9-vghgc"]
Oct 31 03:56:06.544562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.544531    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:56:06.545258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.545232    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns2cp\" (UniqueName: \"kubernetes.io/projected/f6e8985f-f4fe-45bd-a46d-1dacb4865275-kube-api-access-ns2cp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.645812 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.645775    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\") pod \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") "
Oct 31 03:56:06.646003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.645827    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\") pod \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") "
Oct 31 03:56:06.646003 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.645971    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\" (UniqueName: \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\") pod \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") "
Oct 31 03:56:06.646085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.646007    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\") pod \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\" (UID: \"07c8d566-0a7f-43b7-bb17-cfe678e2ebae\") "
Oct 31 03:56:06.646260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.646233    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca" pod "07c8d566-0a7f-43b7-bb17-cfe678e2ebae" (UID: "07c8d566-0a7f-43b7-bb17-cfe678e2ebae"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:06.646798 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.646588    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.648522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.648490    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs" pod "07c8d566-0a7f-43b7-bb17-cfe678e2ebae" (UID: "07c8d566-0a7f-43b7-bb17-cfe678e2ebae"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:06.648661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.648525    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls" pod "07c8d566-0a7f-43b7-bb17-cfe678e2ebae" (UID: "07c8d566-0a7f-43b7-bb17-cfe678e2ebae"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:06.648661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.648566    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq" pod "07c8d566-0a7f-43b7-bb17-cfe678e2ebae" (UID: "07c8d566-0a7f-43b7-bb17-cfe678e2ebae"). InnerVolumeSpecName "kube-api-access-57pcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:06.748058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.747974    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.748058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.748025    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57pcq\" (UniqueName: \"kubernetes.io/projected/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-kube-api-access-57pcq\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:06.748058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:06.748037    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/07c8d566-0a7f-43b7-bb17-cfe678e2ebae-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:07.225089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.224998    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:56:07.226737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.226692    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:56:07.229466 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.229428    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:56:07.236952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.236904    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:56:07.237388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.237356    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:56:07.239131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.239101    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:56:07.347408 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.347382    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"
Oct 31 03:56:07.347899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.347411    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr" event={"ID":"07c8d566-0a7f-43b7-bb17-cfe678e2ebae","Type":"ContainerDied","Data":"44f4901b21f344d562f517e6919ab53edf164ae49b7336ceadbf8d292fecad86"}
Oct 31 03:56:07.347899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.347471    5098 scope.go:122] "RemoveContainer" containerID="60b4c6eb6512954fd98425d854723a838dce877c6f0cbd11fd0a5e619b2901d3"
Oct 31 03:56:07.349051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.348831    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"
Oct 31 03:56:07.409115 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.409073    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:56:07.421867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.421825    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-63258b-5cc8967669-n5fcr"]
Oct 31 03:56:07.446653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.445169    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:56:07.450660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.450628    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-86a834-64f699c64c-mrvx2"]
Oct 31 03:56:07.613532 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.613494    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="001a5098-8bf4-4522-9807-e9db182c5b9c" path="/var/lib/kubelet/pods/001a5098-8bf4-4522-9807-e9db182c5b9c/volumes"
Oct 31 03:56:07.614100 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.614083    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" path="/var/lib/kubelet/pods/07c8d566-0a7f-43b7-bb17-cfe678e2ebae/volumes"
Oct 31 03:56:07.614533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.614517    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" path="/var/lib/kubelet/pods/528eb2c5-8091-4f0c-b8a5-cb98a68e255f/volumes"
Oct 31 03:56:07.615249 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.615233    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" path="/var/lib/kubelet/pods/b1bfcaa4-9246-4137-b4d4-83ab590f40b8/volumes"
Oct 31 03:56:07.615693 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.615677    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" path="/var/lib/kubelet/pods/f6e8985f-f4fe-45bd-a46d-1dacb4865275/volumes"
Oct 31 03:56:07.962670 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.962548    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:56:07.968983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:07.968953    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:56:08.762554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:08.762518    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:56:09.058657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:09.058595    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"]
Oct 31 03:56:09.067852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:09.067794    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-86a834-hfnkw"]
Oct 31 03:56:09.614623 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:09.614580    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c" path="/var/lib/kubelet/pods/44c1ce5b-c58c-4c11-a4cc-9084d2d83b5c/volumes"
Oct 31 03:56:10.199967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:10.199919    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"]
Oct 31 03:56:10.230633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:10.224908    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68780-rwzpg"]
Oct 31 03:56:11.620490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:11.620451    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373fdb48-6cdb-4e1e-8f78-37939255caac" path="/var/lib/kubelet/pods/373fdb48-6cdb-4e1e-8f78-37939255caac/volumes"
Oct 31 03:56:12.093553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:12.091715    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"]
Oct 31 03:56:12.093553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:12.092954    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ce9b89-ctqjs"]
Oct 31 03:56:12.963782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:12.963735    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" podUID="77baa062-a718-4466-9847-1ea715d3e567" containerName="payload-processing" containerID="cri-o://1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1" gracePeriod=25
Oct 31 03:56:12.970212 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:12.970180    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" podUID="8636c4f5-1018-4f64-bf8c-7c7759097338" containerName="payload-pre-processing" containerID="cri-o://741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214" gracePeriod=25
Oct 31 03:56:13.371208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.370871    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:56:13.393110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393080    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:13.393807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393793    5098 cpu_manager.go:414] "containerMap: removing container" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393809    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393822    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b100944e-7e6f-4e9f-8027-215801affb6c" containerName="revoke-keys"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393828    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b100944e-7e6f-4e9f-8027-215801affb6c" containerName="revoke-keys"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393838    5098 cpu_manager.go:414] "containerMap: removing container" podUID="373fdb48-6cdb-4e1e-8f78-37939255caac" containerName="revoke-keys"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393842    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373fdb48-6cdb-4e1e-8f78-37939255caac" containerName="revoke-keys"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393859    5098 cpu_manager.go:414] "containerMap: removing container" podUID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" containerName="maas-api"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393863    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" containerName="maas-api"
Oct 31 03:56:13.393881 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393884    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393889    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393913    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393918    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393927    5098 cpu_manager.go:414] "containerMap: removing container" podUID="77baa062-a718-4466-9847-1ea715d3e567" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393932    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77baa062-a718-4466-9847-1ea715d3e567" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393948    5098 cpu_manager.go:414] "containerMap: removing container" podUID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" containerName="payload-pre-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.393952    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" containerName="payload-pre-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394060    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1bfcaa4-9246-4137-b4d4-83ab590f40b8" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394070    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="77baa062-a718-4466-9847-1ea715d3e567" containerName="payload-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394078    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="528eb2c5-8091-4f0c-b8a5-cb98a68e255f" containerName="payload-pre-processing"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394086    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="252d74ea-9921-4058-8ca6-1d27167967fa" containerName="istio-proxy"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394092    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b100944e-7e6f-4e9f-8027-215801affb6c" containerName="revoke-keys"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394099    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="373fdb48-6cdb-4e1e-8f78-37939255caac" containerName="revoke-keys"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394105    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="07c8d566-0a7f-43b7-bb17-cfe678e2ebae" containerName="maas-api"
Oct 31 03:56:13.394171 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.394112    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6e8985f-f4fe-45bd-a46d-1dacb4865275" containerName="payload-processing"
Oct 31 03:56:13.398067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.398029    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.411678 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.411578    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-86a40d-openshift-default-dockercfg-49mvt\""
Oct 31 03:56:13.416103 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.416066    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:56:13.435054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.435023    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:13.466122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.466072    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\" (UniqueName: \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\") pod \"77baa062-a718-4466-9847-1ea715d3e567\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") "
Oct 31 03:56:13.466372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.466348    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\") pod \"77baa062-a718-4466-9847-1ea715d3e567\" (UID: \"77baa062-a718-4466-9847-1ea715d3e567\") "
Oct 31 03:56:13.467814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.467454    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume" pod "77baa062-a718-4466-9847-1ea715d3e567" (UID: "77baa062-a718-4466-9847-1ea715d3e567"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:13.470699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.470671    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc" pod "77baa062-a718-4466-9847-1ea715d3e567" (UID: "77baa062-a718-4466-9847-1ea715d3e567"). InnerVolumeSpecName "kube-api-access-jfsqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:13.480787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.480741    5098 generic.go:356] "Generic (PLEG): container finished" podID="8636c4f5-1018-4f64-bf8c-7c7759097338" containerID="741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214" exitCode=0
Oct 31 03:56:13.481373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.480831    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" event={"ID":"8636c4f5-1018-4f64-bf8c-7c7759097338","Type":"ContainerDied","Data":"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"}
Oct 31 03:56:13.481373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.480888    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c" event={"ID":"8636c4f5-1018-4f64-bf8c-7c7759097338","Type":"ContainerDied","Data":"5f96bea67b84af039311d05e129b09689ae439899a93b1439ec937d9fe8a22d4"}
Oct 31 03:56:13.481373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.480913    5098 scope.go:122] "RemoveContainer" containerID="741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"
Oct 31 03:56:13.494891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.494859    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"
Oct 31 03:56:13.498563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.497207    5098 generic.go:356] "Generic (PLEG): container finished" podID="77baa062-a718-4466-9847-1ea715d3e567" containerID="1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1" exitCode=0
Oct 31 03:56:13.498563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.497399    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" event={"ID":"77baa062-a718-4466-9847-1ea715d3e567","Type":"ContainerDied","Data":"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"}
Oct 31 03:56:13.498563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.497453    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765" event={"ID":"77baa062-a718-4466-9847-1ea715d3e567","Type":"ContainerDied","Data":"517283fa51078e5247ba0f411cb7dad2a84fc2a839d967ded771da736872812a"}
Oct 31 03:56:13.498563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.497534    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"
Oct 31 03:56:13.515039 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.514908    5098 scope.go:122] "RemoveContainer" containerID="741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"
Oct 31 03:56:13.515397 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:13.515345    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214\": container with ID starting with 741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214 not found: ID does not exist" containerID="741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"
Oct 31 03:56:13.515501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.515407    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214"} err="failed to get container status \"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214\": rpc error: code = NotFound desc = could not find container \"741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214\": container with ID starting with 741976d648b9dfd1c64ba151f751aaf850cad6f4fbb555f4ad605f55b7369214 not found: ID does not exist"
Oct 31 03:56:13.515501 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.515425    5098 scope.go:122] "RemoveContainer" containerID="1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"
Oct 31 03:56:13.535884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.535854    5098 scope.go:122] "RemoveContainer" containerID="1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"
Oct 31 03:56:13.536401 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:13.536300    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1\": container with ID starting with 1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1 not found: ID does not exist" containerID="1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"
Oct 31 03:56:13.536401 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.536363    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1"} err="failed to get container status \"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1\": rpc error: code = NotFound desc = could not find container \"1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1\": container with ID starting with 1c65c8538928d23d6d1667f76f578cc171ea8409d3f35d3739b7c36800e26de1 not found: ID does not exist"
Oct 31 03:56:13.553703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.550849    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:56:13.561521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.561487    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ce9b89-5c9485d4f5-8d765"]
Oct 31 03:56:13.565314 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.565282    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:56:13.565526 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.565500    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy" containerID="cri-o://9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b" gracePeriod=30
Oct 31 03:56:13.567665 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.567574    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\") pod \"8636c4f5-1018-4f64-bf8c-7c7759097338\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") "
Oct 31 03:56:13.567838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.567811    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\" (UniqueName: \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\") pod \"8636c4f5-1018-4f64-bf8c-7c7759097338\" (UID: \"8636c4f5-1018-4f64-bf8c-7c7759097338\") "
Oct 31 03:56:13.568057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.568027    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume" pod "8636c4f5-1018-4f64-bf8c-7c7759097338" (UID: "8636c4f5-1018-4f64-bf8c-7c7759097338"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:13.568725 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.568700    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.568933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.568913    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.569126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.569109    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2scf\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.569353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.569335    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.569822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.569800    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.570113 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570087    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.570241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570213    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.570287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570263    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.570335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570291    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.570515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570494    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77baa062-a718-4466-9847-1ea715d3e567-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:13.570575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570523    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8636c4f5-1018-4f64-bf8c-7c7759097338-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:13.570575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570539    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfsqc\" (UniqueName: \"kubernetes.io/projected/77baa062-a718-4466-9847-1ea715d3e567-kube-api-access-jfsqc\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:13.570944 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.570918    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8" pod "8636c4f5-1018-4f64-bf8c-7c7759097338" (UID: "8636c4f5-1018-4f64-bf8c-7c7759097338"). InnerVolumeSpecName "kube-api-access-9njm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:13.612895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.612861    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77baa062-a718-4466-9847-1ea715d3e567" path="/var/lib/kubelet/pods/77baa062-a718-4466-9847-1ea715d3e567/volumes"
Oct 31 03:56:13.613463 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.613446    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b100944e-7e6f-4e9f-8027-215801affb6c" path="/var/lib/kubelet/pods/b100944e-7e6f-4e9f-8027-215801affb6c/volumes"
Oct 31 03:56:13.679638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.676253    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.685923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.685884    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.686336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.686298    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.693908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.693871    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.694136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.694113    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.694319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.694303    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.694514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.694499    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.694650    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2scf\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.700870    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.701074    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9njm8\" (UniqueName: \"kubernetes.io/projected/8636c4f5-1018-4f64-bf8c-7c7759097338-kube-api-access-9njm8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.697286    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.698977    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.686986    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.701404    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.711646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.697101    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.719199 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.719164    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.728650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.724931    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.732654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.729474    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.734387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.734319    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2scf\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\") pod \"e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") " pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:13.770011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.769874    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" podUID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" containerName="payload-pre-processing" containerID="cri-o://c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6" gracePeriod=25
Oct 31 03:56:13.891288 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.891242    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:56:13.900221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:13.900175    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ce9b89-6fd79486f5-vf46c"]
Oct 31 03:56:14.063538 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.063249    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:14.132829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.132796    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:56:14.219228 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.219192    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"]
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.220736    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8636c4f5-1018-4f64-bf8c-7c7759097338" containerName="payload-pre-processing"
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.220765    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8636c4f5-1018-4f64-bf8c-7c7759097338" containerName="payload-pre-processing"
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.220858    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" containerName="payload-pre-processing"
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.220868    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" containerName="payload-pre-processing"
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.221037    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8636c4f5-1018-4f64-bf8c-7c7759097338" containerName="payload-pre-processing"
Oct 31 03:56:14.224622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.221066    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" containerName="payload-pre-processing"
Oct 31 03:56:14.225732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.225707    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.229192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.229169    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"]
Oct 31 03:56:14.312448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.312416    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:14.312648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.312538    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:56:14.315190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.315125    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\") pod \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") "
Oct 31 03:56:14.315259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.315243    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\" (UniqueName: \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\") pod \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\" (UID: \"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5\") "
Oct 31 03:56:14.315545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.315517    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume" pod "c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" (UID: "c65f9d0b-1d73-47ce-8cd1-220b1b1376d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:14.316040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.316023    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:14.317347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.317328    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256" pod "c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" (UID: "c65f9d0b-1d73-47ce-8cd1-220b1b1376d5"). InnerVolumeSpecName "kube-api-access-pk256". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:14.386213 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:14.386180    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3ddb675_e819_4515_9606_ef5b0dcd60e0.slice/crio-ebcaef113aabb26804f362f3025fc00c61e1151493b16c2b4fafd86f3e73f32a WatchSource:0}: Error finding container ebcaef113aabb26804f362f3025fc00c61e1151493b16c2b4fafd86f3e73f32a: Status 404 returned error can't find the container with id ebcaef113aabb26804f362f3025fc00c61e1151493b16c2b4fafd86f3e73f32a
Oct 31 03:56:14.388067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.388040    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:14.389516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.389475    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:14.389583 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.389567    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:14.389634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.389623    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:14.416927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.416894    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.417178 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.417159    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rz8z\" (UniqueName: \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.417232 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.417202    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.417381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.417367    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk256\" (UniqueName: \"kubernetes.io/projected/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5-kube-api-access-pk256\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:14.509178 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.509141    5098 generic.go:356] "Generic (PLEG): container finished" podID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" containerID="c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6" exitCode=0
Oct 31 03:56:14.509359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.509223    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"
Oct 31 03:56:14.509359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.509227    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" event={"ID":"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5","Type":"ContainerDied","Data":"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"}
Oct 31 03:56:14.509359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.509294    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf" event={"ID":"c65f9d0b-1d73-47ce-8cd1-220b1b1376d5","Type":"ContainerDied","Data":"ad3cc1c592f493bc52e100df129a0b515097cade8419bf462a4d55305c8555a9"}
Oct 31 03:56:14.509359 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.509323    5098 scope.go:122] "RemoveContainer" containerID="c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.517460    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" event={"ID":"c3ddb675-e819-4515-9606-ef5b0dcd60e0","Type":"ContainerStarted","Data":"c2e7e53e170aeb3e6b02e36c4cf59ab5a44faa59e944e28ded976826ec1e6182"}
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.517517    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" event={"ID":"c3ddb675-e819-4515-9606-ef5b0dcd60e0","Type":"ContainerStarted","Data":"ebcaef113aabb26804f362f3025fc00c61e1151493b16c2b4fafd86f3e73f32a"}
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.518858    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rz8z\" (UniqueName: \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.518903    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.519042    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.520128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.520057    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.523243 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.523221    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.532366 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.532341    5098 scope.go:122] "RemoveContainer" containerID="c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"
Oct 31 03:56:14.532724 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:14.532701    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6\": container with ID starting with c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6 not found: ID does not exist" containerID="c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"
Oct 31 03:56:14.532838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.532731    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6"} err="failed to get container status \"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6\": rpc error: code = NotFound desc = could not find container \"c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6\": container with ID starting with c71dc0d6e7c210c4049ba52dfe06832c1cca5bb07622baa0d3d6e131d693ced6 not found: ID does not exist"
Oct 31 03:56:14.532838 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.532761    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rz8z\" (UniqueName: \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\") pod \"maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.546398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.546371    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:14.548974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.548911    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podStartSLOduration=1.548892691 podStartE2EDuration="1.548892691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:14.543874724 +0000 UTC m=+2449.069094350" watchObservedRunningTime="2026-10-31 03:56:14.548892691 +0000 UTC m=+2449.074112281"
Oct 31 03:56:14.565033 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.565004    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:56:14.569955 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.569928    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b68780-7fbb57784c-hgdhf"]
Oct 31 03:56:14.849162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:14.849129    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"]
Oct 31 03:56:14.849358 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:14.849334    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd061516_8162_4c3e_b970_e0b967ca65b8.slice/crio-f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722 WatchSource:0}: Error finding container f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722: Status 404 returned error can't find the container with id f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722
Oct 31 03:56:15.064905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.064866    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:15.066731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.066702    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:15.066869 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.066767    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:15.528150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.528114    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv" event={"ID":"cd061516-8162-4c3e-b970-e0b967ca65b8","Type":"ContainerStarted","Data":"9723dee303dd842a18cb2b95f9105b4108f3e1cfdb0d21ebcd72cf465007cf8d"}
Oct 31 03:56:15.528326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.528175    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv" event={"ID":"cd061516-8162-4c3e-b970-e0b967ca65b8","Type":"ContainerStarted","Data":"f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722"}
Oct 31 03:56:15.579106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.579048    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv" podStartSLOduration=1.579032574 podStartE2EDuration="1.579032574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:15.574967977 +0000 UTC m=+2450.100187575" watchObservedRunningTime="2026-10-31 03:56:15.579032574 +0000 UTC m=+2450.104252162"
Oct 31 03:56:15.613752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.613717    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8636c4f5-1018-4f64-bf8c-7c7759097338" path="/var/lib/kubelet/pods/8636c4f5-1018-4f64-bf8c-7c7759097338/volumes"
Oct 31 03:56:15.614315 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:15.614296    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65f9d0b-1d73-47ce-8cd1-220b1b1376d5" path="/var/lib/kubelet/pods/c65f9d0b-1d73-47ce-8cd1-220b1b1376d5/volumes"
Oct 31 03:56:16.067373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.066447    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:16.067373 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.066506    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:16.099724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.099679    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.123740    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.125245    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.129156    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-20585c\"/\"kube-root-ca.crt\""
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.129296    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-20585c\"/\"auth-test-model-20585c-kserve-self-signed-certs\""
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.129431    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w6-20585c\"/\"default-dockercfg-4k4pd\""
Oct 31 03:56:16.134708 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.129508    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w6-20585c\"/\"openshift-service-ca.crt\""
Oct 31 03:56:16.196923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.196878    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.197285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.197169    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.197386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.197319    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.197548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.197434    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25l5q\" (UniqueName: \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.197640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.197601    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.197709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.197691    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.303850    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.303929    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.303961    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.304087    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25l5q\" (UniqueName: \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.304143    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.304262    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.313629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.304985    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.324673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.320787    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.324673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.321038    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.326883 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.326384    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.340630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.336888    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.349646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.348917    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25l5q\" (UniqueName: \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\") pod \"auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") " pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.519861 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.519826    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:16.541119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.541077    5098 generic.go:356] "Generic (PLEG): container finished" podID="cd061516-8162-4c3e-b970-e0b967ca65b8" containerID="9723dee303dd842a18cb2b95f9105b4108f3e1cfdb0d21ebcd72cf465007cf8d" exitCode=0
Oct 31 03:56:16.541295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.541159    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv" event={"ID":"cd061516-8162-4c3e-b970-e0b967ca65b8","Type":"ContainerDied","Data":"9723dee303dd842a18cb2b95f9105b4108f3e1cfdb0d21ebcd72cf465007cf8d"}
Oct 31 03:56:16.831676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.820790    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:56:16.831676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:16.821050    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" containerID="cri-o://2648e7ea608ba401cfaff26ed57758ec1d68246db5c3b65d8e8fbddf78d2ce5f" gracePeriod=30
Oct 31 03:56:17.115636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.114916    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:17.115636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.114980    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:17.153979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.142330    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:56:17.577262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.577220    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" event={"ID":"67f732fb-d44d-467d-942e-11431c60f267","Type":"ContainerStarted","Data":"130c68a23c8a52615f53d944b35ed7672802ee1a48632d2757c081be90e839f3"}
Oct 31 03:56:17.577482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.577276    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" event={"ID":"67f732fb-d44d-467d-942e-11431c60f267","Type":"ContainerStarted","Data":"1727edeae732654e07366d32cfd0415191def3a8518a5f06c205b4484ae0d5d5"}
Oct 31 03:56:17.608194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.607228    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" podStartSLOduration=1.607209315 podStartE2EDuration="1.607209315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:17.60024192 +0000 UTC m=+2452.125461509" watchObservedRunningTime="2026-10-31 03:56:17.607209315 +0000 UTC m=+2452.132428905"
Oct 31 03:56:17.818677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.818574    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"]
Oct 31 03:56:17.828540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.825733    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:17.832935 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.831694    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"]
Oct 31 03:56:17.905089 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.904835    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:17.910029 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.909968    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.910777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.910755    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:17.915262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915237    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\" (UniqueName: \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\") pod \"cd061516-8162-4c3e-b970-e0b967ca65b8\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") "
Oct 31 03:56:17.915789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915313    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\") pod \"cd061516-8162-4c3e-b970-e0b967ca65b8\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") "
Oct 31 03:56:17.915789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915424    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\") pod \"cd061516-8162-4c3e-b970-e0b967ca65b8\" (UID: \"cd061516-8162-4c3e-b970-e0b967ca65b8\") "
Oct 31 03:56:17.915789 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915759    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915798    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915832    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915866    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpz7h\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915896    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.915992    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76k74\" (UniqueName: \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:17.916047 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916043    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916070    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca" pod "cd061516-8162-4c3e-b970-e0b967ca65b8" (UID: "cd061516-8162-4c3e-b970-e0b967ca65b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:17.916357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916096    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916209    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916351    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916405    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:17.916518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916473    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:17.916714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.916559    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:17.918073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.918048    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ce9b89-openshift-default-dockercfg-zsljk\""
Oct 31 03:56:17.918492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.918466    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca" pod "cd061516-8162-4c3e-b970-e0b967ca65b8" (UID: "cd061516-8162-4c3e-b970-e0b967ca65b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:17.919329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.919305    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z" pod "cd061516-8162-4c3e-b970-e0b967ca65b8" (UID: "cd061516-8162-4c3e-b970-e0b967ca65b8"). InnerVolumeSpecName "kube-api-access-2rz8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:17.920301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:17.920277    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:18.018023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.017984    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018050    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018120    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018206    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018239    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018267    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018294    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpz7h\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018327    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018440    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76k74\" (UniqueName: \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018499    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018546    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018561    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018636    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018791    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rz8z\" (UniqueName: \"kubernetes.io/projected/cd061516-8162-4c3e-b970-e0b967ca65b8-kube-api-access-2rz8z\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:18.018988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018794    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.018988 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.018811    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd061516-8162-4c3e-b970-e0b967ca65b8-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:18.019873 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.019842    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.019999 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.019898    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.020139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.020114    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.020755 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.020727    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.021831 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.021800    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.022321 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.022291    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.022728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.022576    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.028718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.028680    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76k74\" (UniqueName: \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\") pod \"maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.028851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.028800    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.029208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.029183    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpz7h\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\") pod \"e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") " pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.074156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.074110    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:18.074400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.074341    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:18.209407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.209322    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:18.238129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.236559    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:18.342721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.342675    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:56:18.388694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.343069    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" containerID="cri-o://1523f4d50f9ccf3bc50f38ec902426576925c93fd3221afd74cb6fdeeb8d2886" gracePeriod=30
Oct 31 03:56:18.573765 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.523000    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:18.799339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.797745    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:56:18.799339 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.798132    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" containerID="cri-o://6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4" gracePeriod=30
Oct 31 03:56:18.829841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.829685    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"
Oct 31 03:56:18.829841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.829730    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv" event={"ID":"cd061516-8162-4c3e-b970-e0b967ca65b8","Type":"ContainerDied","Data":"f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722"}
Oct 31 03:56:18.829841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.829789    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f12cf1c64a37ac468cd7c5a10219b32dd8336c1d64090361ab7071775694b722"
Oct 31 03:56:18.836791 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.836702    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"]
Oct 31 03:56:18.837647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.837473    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cd061516-8162-4c3e-b970-e0b967ca65b8" containerName="revoke-keys"
Oct 31 03:56:18.837647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.837502    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd061516-8162-4c3e-b970-e0b967ca65b8" containerName="revoke-keys"
Oct 31 03:56:18.838506 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.837950    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd061516-8162-4c3e-b970-e0b967ca65b8" containerName="revoke-keys"
Oct 31 03:56:18.873283 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.873255    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:18.894683 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.894641    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"]
Oct 31 03:56:18.939585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.938549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:18.939585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.938762    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:18.939585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:18.938891    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.030358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.030325    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:56:19.034645 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:19.034570    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5ab9f0_1aea_422f_a24e_8ae51101dd99.slice/crio-e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1 WatchSource:0}: Error finding container e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1: Status 404 returned error can't find the container with id e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1
Oct 31 03:56:19.041775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.040269    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.041775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.040404    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.041775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.040526    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.042684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.042420    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.042684 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.042593    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"]
Oct 31 03:56:19.082640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.081327    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.082640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.084907    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:19.086320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.085557    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:19.086320 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.085672    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:19.088475 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.087238    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\") pod \"maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.092948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.092914    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:19.093081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.092979    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:19.093081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.093010    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:56:19.212428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.211887    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:19.239650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.239629    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.391137    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.391270    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.391372    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.391413    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.391451    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.392089    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.392468    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.392742    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.395914    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.395974    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.396085    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.396150    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\") pod \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\" (UID: \"138a0e1e-19ac-4d88-81f4-a5334a9fbc81\") "
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.396900    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397059    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397086    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397102    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397117    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397117    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:19.401418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.397299    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:19.405537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.401877    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:19.405537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.402010    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:19.405537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.404882    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9" pod "138a0e1e-19ac-4d88-81f4-a5334a9fbc81" (UID: "138a0e1e-19ac-4d88-81f4-a5334a9fbc81"). InnerVolumeSpecName "kube-api-access-66lj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:19.607674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.521592    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66lj9\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-kube-api-access-66lj9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.607674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.521661    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.607674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.521680    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.607674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.521694    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.607674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.521710    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138a0e1e-19ac-4d88-81f4-a5334a9fbc81-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:19.712571 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:19.712370    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e0ffc4_1d10_4a97_a95b_bd24c2150798.slice/crio-ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64 WatchSource:0}: Error finding container ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64: Status 404 returned error can't find the container with id ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64
Oct 31 03:56:19.789714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.787670    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"]
Oct 31 03:56:19.841836 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.841797    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n" event={"ID":"4d5ab9f0-1aea-422f-a24e-8ae51101dd99","Type":"ContainerStarted","Data":"ad888ad844a4fd29e11e6c1099d0ef1ab09cb779229fdd77dd4a70471a98db90"}
Oct 31 03:56:19.842043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.841865    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n" event={"ID":"4d5ab9f0-1aea-422f-a24e-8ae51101dd99","Type":"ContainerStarted","Data":"e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1"}
Oct 31 03:56:19.843402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.843370    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl" event={"ID":"d0e0ffc4-1d10-4a97-a95b-bd24c2150798","Type":"ContainerStarted","Data":"ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64"}
Oct 31 03:56:19.845497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.845462    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" event={"ID":"1aaad8d5-3920-44c9-baa9-a0e1674f5e08","Type":"ContainerStarted","Data":"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"}
Oct 31 03:56:19.845627 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.845537    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" event={"ID":"1aaad8d5-3920-44c9-baa9-a0e1674f5e08","Type":"ContainerStarted","Data":"05ac0c21cf0cb0aaa0ee6e426b2e8c4beb8f94ae341a6eedb012bdeb8613eec6"}
Oct 31 03:56:19.847285 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.847253    5098 generic.go:356] "Generic (PLEG): container finished" podID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerID="9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b" exitCode=0
Oct 31 03:56:19.847386 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.847300    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"
Oct 31 03:56:19.847468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.847389    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" event={"ID":"138a0e1e-19ac-4d88-81f4-a5334a9fbc81","Type":"ContainerDied","Data":"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"}
Oct 31 03:56:19.847468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.847440    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh" event={"ID":"138a0e1e-19ac-4d88-81f4-a5334a9fbc81","Type":"ContainerDied","Data":"871b1de531b257f46b9f57dd9a2a91372e9c9ecf3c4b3cdfcfc0c7051d2165ea"}
Oct 31 03:56:19.847550 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.847469    5098 scope.go:122] "RemoveContainer" containerID="9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"
Oct 31 03:56:19.894843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.894810    5098 scope.go:122] "RemoveContainer" containerID="9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"
Oct 31 03:56:19.946302 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:19.945069    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b\": container with ID starting with 9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b not found: ID does not exist" containerID="9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"
Oct 31 03:56:19.946302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.945119    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b"} err="failed to get container status \"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b\": rpc error: code = NotFound desc = could not find container \"9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b\": container with ID starting with 9b54c57c01ffc803f70c2013022de68f7068230bc8027a9bc874ffcd8c7c844b not found: ID does not exist"
Oct 31 03:56:19.976636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:19.963562    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n" podStartSLOduration=2.963539072 podStartE2EDuration="2.963539072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:19.862040812 +0000 UTC m=+2454.387260406" watchObservedRunningTime="2026-10-31 03:56:19.963539072 +0000 UTC m=+2454.488758662"
Oct 31 03:56:20.019076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.019041    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:56:20.066772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.066737    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:20.067014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.066988    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:20.136274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.136241    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-63258b-openshift-default-7cb66d87b9-8q5rh"]
Oct 31 03:56:20.144839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.144776    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podStartSLOduration=3.144756256 podStartE2EDuration="3.144756256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:20.019015296 +0000 UTC m=+2454.544234862" watchObservedRunningTime="2026-10-31 03:56:20.144756256 +0000 UTC m=+2454.669975847"
Oct 31 03:56:20.171695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.167447    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"]
Oct 31 03:56:20.171695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.168826    5098 cpu_manager.go:414] "containerMap: removing container" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy"
Oct 31 03:56:20.171695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.168851    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy"
Oct 31 03:56:20.171695 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.169045    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" containerName="istio-proxy"
Oct 31 03:56:20.174381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.174355    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.195372 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.194667    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"]
Oct 31 03:56:20.238044 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.237930    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:20.239335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.239301    5098 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 31 03:56:20.239488 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.239362    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 31 03:56:20.277767 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.277728    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97pln\" (UniqueName: \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.277910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.277832    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.278392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.278134    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.380116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.380076    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97pln\" (UniqueName: \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.380335 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.380150    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.380478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.380452    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.381215 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.381188    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.385749 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.385670    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.390132 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.390097    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97pln\" (UniqueName: \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\") pod \"maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:20.505718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:20.505590    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:21.083121 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.082626    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:21.083121 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.082690    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:21.132718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.132679    5098 generic.go:356] "Generic (PLEG): container finished" podID="4d5ab9f0-1aea-422f-a24e-8ae51101dd99" containerID="ad888ad844a4fd29e11e6c1099d0ef1ab09cb779229fdd77dd4a70471a98db90" exitCode=0
Oct 31 03:56:21.132931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.132771    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n" event={"ID":"4d5ab9f0-1aea-422f-a24e-8ae51101dd99","Type":"ContainerDied","Data":"ad888ad844a4fd29e11e6c1099d0ef1ab09cb779229fdd77dd4a70471a98db90"}
Oct 31 03:56:21.134894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.134861    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl" event={"ID":"d0e0ffc4-1d10-4a97-a95b-bd24c2150798","Type":"ContainerStarted","Data":"274dc06924257190f7f7752a7b42a470244909a3b3ac30cadc9ce42159638126"}
Oct 31 03:56:21.243666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.241009    5098 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 31 03:56:21.243666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.241073    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 31 03:56:21.479120 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.454896    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl" podStartSLOduration=3.454873628 podStartE2EDuration="3.454873628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:21.368677174 +0000 UTC m=+2455.893896765" watchObservedRunningTime="2026-10-31 03:56:21.454873628 +0000 UTC m=+2455.980093215"
Oct 31 03:56:21.948012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.911582    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection reset by peer" start-of-body=
Oct 31 03:56:21.948012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:21.911713    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection reset by peer"
Oct 31 03:56:22.109965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.076579    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138a0e1e-19ac-4d88-81f4-a5334a9fbc81" path="/var/lib/kubelet/pods/138a0e1e-19ac-4d88-81f4-a5334a9fbc81/volumes"
Oct 31 03:56:22.109965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.077204    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"]
Oct 31 03:56:22.109965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.078147    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:22.109965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.078198    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:22.403632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.360386    5098 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 31 03:56:22.448632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.435983    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 31 03:56:22.520420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.464787    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:56:22.718775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.718677    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:22.725141 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.725082    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:56:22.726090 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.726058    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:56:22.731464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.731427    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy" containerID="cri-o://579f511660a1fd83d64101cff99bfcafd3a1266fc6765bcaadafd2e442771351" gracePeriod=30
Oct 31 03:56:22.741718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.741672    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:56:22.742374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.742332    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy" containerID="cri-o://7352a298d00977dff2344efa58357f229435e807df2e5c90355882ba52253506" gracePeriod=30
Oct 31 03:56:22.785332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.784747    5098 generic.go:356] "Generic (PLEG): container finished" podID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerID="2648e7ea608ba401cfaff26ed57758ec1d68246db5c3b65d8e8fbddf78d2ce5f" exitCode=0
Oct 31 03:56:22.785332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.784980    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerDied","Data":"2648e7ea608ba401cfaff26ed57758ec1d68246db5c3b65d8e8fbddf78d2ce5f"}
Oct 31 03:56:22.785332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.785051    5098 scope.go:122] "RemoveContainer" containerID="5e36d47f8fcbc6170d63b1bbc61552c27a1bad8e6fde9a95e5b60cd8f49ffc66"
Oct 31 03:56:22.792909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.792714    5098 generic.go:356] "Generic (PLEG): container finished" podID="d0e0ffc4-1d10-4a97-a95b-bd24c2150798" containerID="274dc06924257190f7f7752a7b42a470244909a3b3ac30cadc9ce42159638126" exitCode=0
Oct 31 03:56:22.792909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.792833    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl" event={"ID":"d0e0ffc4-1d10-4a97-a95b-bd24c2150798","Type":"ContainerDied","Data":"274dc06924257190f7f7752a7b42a470244909a3b3ac30cadc9ce42159638126"}
Oct 31 03:56:22.797497 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.796915    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg" event={"ID":"470e0e9e-6fff-4aae-a82f-323f0a14bcbc","Type":"ContainerStarted","Data":"8f4987eb7d1a4c4ff73daf49089869105a2222eaad8a350739fc09e42aa1c9b5"}
Oct 31 03:56:22.797844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.797805    5098 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" secret="" err="secret \"e2e-inf-b-ce9b89-openshift-default-dockercfg-zsljk\" not found"
Oct 31 03:56:22.829640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.820489    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:56:22.829640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.821090    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" containerID="cri-o://687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701" gracePeriod=30
Oct 31 03:56:22.829640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.828950    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"]
Oct 31 03:56:22.845739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.844666    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:22.847323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.844550    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"]
Oct 31 03:56:22.896984 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.896949    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"]
Oct 31 03:56:22.913024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.913000    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"]
Oct 31 03:56:22.913318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.913253    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:22.996194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.996097    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:22.996771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.996744    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:22.997575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.997549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr494\" (UniqueName: \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:22.997799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.997780    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:22.997975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.997958    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:22.998209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:22.998192    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.093825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.093582    5098 patch_prober.go:29] interesting pod/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body=
Oct 31 03:56:23.093825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.093706    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104244    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104300    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr494\" (UniqueName: \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104368    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104453    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104655    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.104961 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.104881    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.107173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.106226    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.108854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.108153    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.113462 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.111830    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.123131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.123089    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.125845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.125811    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr494\" (UniqueName: \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.127702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.127530    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.197086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.197054    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:56:23.233191 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.231954    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:23.249859 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.248308    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:23.254854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.253888    5098 patch_prober.go:29] interesting pod/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused" start-of-body=
Oct 31 03:56:23.254854 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.253967    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.25:15021/healthz/ready\": dial tcp 10.132.1.25:15021: connect: connection refused"
Oct 31 03:56:23.309957 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.309906    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310039    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310151 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310104    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310174    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310286    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310443 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310419    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.310540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.310523    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.312280 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.311762    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:23.313580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.313551    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:23.313993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.313949    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:23.315542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.314383    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:23.319758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.318200    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:23.319758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.318948    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:23.323823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.323776    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "kube-api-access-n2j9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:23.329967 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.329937    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:23.418545 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.418508    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.419025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.418623    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\") pod \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\" (UID: \"7ea2218c-5720-4f94-af70-b8c0f5445a9b\") "
Oct 31 03:56:23.419025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.418942    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430580    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430655    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430674    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430690    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430706    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430722    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2j9x\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-kube-api-access-n2j9x\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430739    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.430797 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.430756    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ea2218c-5720-4f94-af70-b8c0f5445a9b-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.441821 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.441769    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token" pod "7ea2218c-5720-4f94-af70-b8c0f5445a9b" (UID: "7ea2218c-5720-4f94-af70-b8c0f5445a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:23.536829 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.536777    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\") pod \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") "
Oct 31 03:56:23.537205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.537108    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\") pod \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") "
Oct 31 03:56:23.537205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.537169    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\" (UniqueName: \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\") pod \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\" (UID: \"4d5ab9f0-1aea-422f-a24e-8ae51101dd99\") "
Oct 31 03:56:23.537205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.537169    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca" pod "4d5ab9f0-1aea-422f-a24e-8ae51101dd99" (UID: "4d5ab9f0-1aea-422f-a24e-8ae51101dd99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:23.538311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.538282    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ea2218c-5720-4f94-af70-b8c0f5445a9b-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.538407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.538316    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.541302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.541269    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74" pod "4d5ab9f0-1aea-422f-a24e-8ae51101dd99" (UID: "4d5ab9f0-1aea-422f-a24e-8ae51101dd99"). InnerVolumeSpecName "kube-api-access-76k74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:23.541405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.541381    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca" pod "4d5ab9f0-1aea-422f-a24e-8ae51101dd99" (UID: "4d5ab9f0-1aea-422f-a24e-8ae51101dd99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:23.643494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.640890    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.643494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.640923    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76k74\" (UniqueName: \"kubernetes.io/projected/4d5ab9f0-1aea-422f-a24e-8ae51101dd99-kube-api-access-76k74\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:23.724080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.724044    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"]
Oct 31 03:56:23.843876 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.843778    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"]
Oct 31 03:56:23.962079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.959124    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf" event={"ID":"7ea2218c-5720-4f94-af70-b8c0f5445a9b","Type":"ContainerDied","Data":"dfa3925811eb1bfb83fe2738fc0c838d81df1de2fdc14dae3ec3125fc6132927"}
Oct 31 03:56:23.962079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.959197    5098 scope.go:122] "RemoveContainer" containerID="2648e7ea608ba401cfaff26ed57758ec1d68246db5c3b65d8e8fbddf78d2ce5f"
Oct 31 03:56:23.962079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.959333    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"
Oct 31 03:56:23.993629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.993569    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp" event={"ID":"5d1e7197-521d-4bf2-9786-eb9a37a84b7d","Type":"ContainerStarted","Data":"5bac5b026d338228c7f00fb8a90848207aa096f2e200826e1b80ee88dec5b5fc"}
Oct 31 03:56:23.998197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.996939    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n" event={"ID":"4d5ab9f0-1aea-422f-a24e-8ae51101dd99","Type":"ContainerDied","Data":"e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1"}
Oct 31 03:56:23.998197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.996998    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e940ab103f24feaf91b248c2404c2acc3958fc4130bf64267579f707bdfa95a1"
Oct 31 03:56:23.998197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:23.997083    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"
Oct 31 03:56:24.002680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.002537    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg" event={"ID":"470e0e9e-6fff-4aae-a82f-323f0a14bcbc","Type":"ContainerStarted","Data":"30e7cbbf0d7a6a7916185f54d4e737cc4f7cc5a0ea812facff47346bdc66541b"}
Oct 31 03:56:24.007676 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.006820    5098 generic.go:356] "Generic (PLEG): container finished" podID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerID="1523f4d50f9ccf3bc50f38ec902426576925c93fd3221afd74cb6fdeeb8d2886" exitCode=0
Oct 31 03:56:24.008258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.008081    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" event={"ID":"6d7936d4-6ad3-4216-990d-d7cc5d26902c","Type":"ContainerDied","Data":"1523f4d50f9ccf3bc50f38ec902426576925c93fd3221afd74cb6fdeeb8d2886"}
Oct 31 03:56:24.008858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.008430    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy" containerID="cri-o://9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674" gracePeriod=30
Oct 31 03:56:24.032301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.032233    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg" podStartSLOduration=4.032210989 podStartE2EDuration="4.032210989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:24.030854443 +0000 UTC m=+2458.556074033" watchObservedRunningTime="2026-10-31 03:56:24.032210989 +0000 UTC m=+2458.557430578"
Oct 31 03:56:24.088511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.084005    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:24.089010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.088977    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:24.090329 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.090308    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:24.137815 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.137376    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:56:24.210969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.210915    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:56:24.223328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.223294    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-86a834-openshift-default-7f55c8dcc-2h5gf"]
Oct 31 03:56:24.285224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285032    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285113    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285224 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285197    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285236    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285336    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285446    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285558    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.285710 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285665    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.286046 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.285757    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\") pod \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\" (UID: \"6d7936d4-6ad3-4216-990d-d7cc5d26902c\") "
Oct 31 03:56:24.287278 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.287253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.288737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.287632    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.288737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.287948    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:24.289075 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.289016    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.290129 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.290104    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.292248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.292201    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "kube-api-access-5dcvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:24.294959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.294916    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:24.295085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.295036    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.295917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.295876    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token" pod "6d7936d4-6ad3-4216-990d-d7cc5d26902c" (UID: "6d7936d4-6ad3-4216-990d-d7cc5d26902c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:24.301509 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.301087    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:56:24.389983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.389955    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390185    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390279 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390267    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390337 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390328    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390396 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390386    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390464 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390456    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcvn\" (UniqueName: \"kubernetes.io/projected/6d7936d4-6ad3-4216-990d-d7cc5d26902c-kube-api-access-5dcvn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390525 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390517    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390591 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390582    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.390718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.390706    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d7936d4-6ad3-4216-990d-d7cc5d26902c-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.408734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.408705    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.491752    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.491805    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.491877    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.491906    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.491945    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492107    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\") pod \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492187    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492264    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492315    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\") pod \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492349    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492412    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\") pod \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\" (UID: \"d0e0ffc4-1d10-4a97-a95b-bd24c2150798\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.492445    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\") pod \"1f404570-36f6-458e-b0cd-b64a3ae53f89\" (UID: \"1f404570-36f6-458e-b0cd-b64a3ae53f89\") "
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.494402    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca" pod "d0e0ffc4-1d10-4a97-a95b-bd24c2150798" (UID: "d0e0ffc4-1d10-4a97-a95b-bd24c2150798"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.495830    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.497063    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.497196    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.497915    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:24.504813 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.498493    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.573781 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.572828    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "kube-api-access-6d7h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:24.579549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.579512    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:24.580207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.580174    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca" pod "d0e0ffc4-1d10-4a97-a95b-bd24c2150798" (UID: "d0e0ffc4-1d10-4a97-a95b-bd24c2150798"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:24.580800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.580767    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz" pod "d0e0ffc4-1d10-4a97-a95b-bd24c2150798" (UID: "d0e0ffc4-1d10-4a97-a95b-bd24c2150798"). InnerVolumeSpecName "kube-api-access-kfntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:24.580956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.580916    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:24.581107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.580958    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo" pod "1f404570-36f6-458e-b0cd-b64a3ae53f89" (UID: "1f404570-36f6-458e-b0cd-b64a3ae53f89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:24.595101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595068    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595101 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595102    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595117    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595126    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-kube-api-access-kfntz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595136    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595144    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e0ffc4-1d10-4a97-a95b-bd24c2150798-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595153    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d7h5\" (UniqueName: \"kubernetes.io/projected/1f404570-36f6-458e-b0cd-b64a3ae53f89-kube-api-access-6d7h5\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595162    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595171    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595185    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f404570-36f6-458e-b0cd-b64a3ae53f89-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595197    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.595325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.595205    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f404570-36f6-458e-b0cd-b64a3ae53f89-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:24.987631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.976987    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:56:24.987631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.979726    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:56:24.987631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:24.979792    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:56:25.034820 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.034739    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp" event={"ID":"5d1e7197-521d-4bf2-9786-eb9a37a84b7d","Type":"ContainerStarted","Data":"9dbcaf19006ca0e65ca00c1823ab65a4dfa0bbb4d0c1ad4be1b6f0ecc198745d"}
Oct 31 03:56:25.036451 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.036429    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"
Oct 31 03:56:25.036720 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.036445    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl" event={"ID":"d0e0ffc4-1d10-4a97-a95b-bd24c2150798","Type":"ContainerDied","Data":"ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64"}
Oct 31 03:56:25.037403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.036746    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee4bc1a96d852305797830a4ea9cfcf95d296f48c27f77733cebdd8ff5a03b64"
Oct 31 03:56:25.038175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.038149    5098 generic.go:356] "Generic (PLEG): container finished" podID="470e0e9e-6fff-4aae-a82f-323f0a14bcbc" containerID="30e7cbbf0d7a6a7916185f54d4e737cc4f7cc5a0ea812facff47346bdc66541b" exitCode=0
Oct 31 03:56:25.038273 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.038219    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg" event={"ID":"470e0e9e-6fff-4aae-a82f-323f0a14bcbc","Type":"ContainerDied","Data":"30e7cbbf0d7a6a7916185f54d4e737cc4f7cc5a0ea812facff47346bdc66541b"}
Oct 31 03:56:25.040323 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.040294    5098 generic.go:356] "Generic (PLEG): container finished" podID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerID="6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4" exitCode=0
Oct 31 03:56:25.040424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.040356    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"
Oct 31 03:56:25.040424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.040410    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" event={"ID":"1f404570-36f6-458e-b0cd-b64a3ae53f89","Type":"ContainerDied","Data":"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"}
Oct 31 03:56:25.040548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.040445    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" event={"ID":"1f404570-36f6-458e-b0cd-b64a3ae53f89","Type":"ContainerDied","Data":"eeeeb8a600011fef5a9ab515092d331ce3cbd6aaad941c92ece71a08033d9ee8"}
Oct 31 03:56:25.040548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.040480    5098 scope.go:122] "RemoveContainer" containerID="6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"
Oct 31 03:56:25.043005 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.042976    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v" event={"ID":"6d7936d4-6ad3-4216-990d-d7cc5d26902c","Type":"ContainerDied","Data":"bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836"}
Oct 31 03:56:25.043118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.043097    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"
Oct 31 03:56:25.096830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.096766    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp" podStartSLOduration=3.096743991 podStartE2EDuration="3.096743991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:25.08523895 +0000 UTC m=+2459.610458540" watchObservedRunningTime="2026-10-31 03:56:25.096743991 +0000 UTC m=+2459.621963580"
Oct 31 03:56:25.103418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.103385    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps" event={"ID":"8aa803d2-44a1-4de3-a49d-b751455fa261","Type":"ContainerStarted","Data":"9284f84ee7c207713cfe7bd9dd48ef95b00bb25495834e13904f95a36a8c3e33"}
Oct 31 03:56:25.103769 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.103745    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps" event={"ID":"8aa803d2-44a1-4de3-a49d-b751455fa261","Type":"ContainerStarted","Data":"13a7974153fb635a4a419ee2a301b2402fb8c4d718a04fdc62593bcd7ad6a071"}
Oct 31 03:56:25.104095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.104076    5098 scope.go:122] "RemoveContainer" containerID="6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"
Oct 31 03:56:25.105336 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:25.105311    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4\": container with ID starting with 6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4 not found: ID does not exist" containerID="6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"
Oct 31 03:56:25.105584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.105483    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4"} err="failed to get container status \"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4\": rpc error: code = NotFound desc = could not find container \"6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4\": container with ID starting with 6b6be2de8dca160dfa91431be1473f0a3443a7f891e604fdeb448c96b73972b4 not found: ID does not exist"
Oct 31 03:56:25.105759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.105589    5098 scope.go:122] "RemoveContainer" containerID="1523f4d50f9ccf3bc50f38ec902426576925c93fd3221afd74cb6fdeeb8d2886"
Oct 31 03:56:25.127462 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:25.127432    5098 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d7936d4_6ad3_4216_990d_d7cc5d26902c.slice/crio-bff000bc4c37a2bed86d4772632ef976b0d79359c2e458ed715c9e5d78f6a836\": RecentStats: unable to find data in memory cache]"
Oct 31 03:56:25.137381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.137350    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:56:25.145299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.145272    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz"]
Oct 31 03:56:25.181266 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.181195    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps" podStartSLOduration=3.181172762 podStartE2EDuration="3.181172762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:25.171730166 +0000 UTC m=+2459.696949756" watchObservedRunningTime="2026-10-31 03:56:25.181172762 +0000 UTC m=+2459.706392353"
Oct 31 03:56:25.193313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.193270    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:56:25.199908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.199876    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-9zg6v"]
Oct 31 03:56:25.234727 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.234684    5098 patch_prober.go:29] interesting pod/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:56:25.234910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.234752    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5d9b6-openshift-default-d48688bb5-z7hpz" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:56:25.614790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.614709    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" path="/var/lib/kubelet/pods/1f404570-36f6-458e-b0cd-b64a3ae53f89/volumes"
Oct 31 03:56:25.615308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.615292    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" path="/var/lib/kubelet/pods/6d7936d4-6ad3-4216-990d-d7cc5d26902c/volumes"
Oct 31 03:56:25.615804 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:25.615789    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" path="/var/lib/kubelet/pods/7ea2218c-5720-4f94-af70-b8c0f5445a9b/volumes"
Oct 31 03:56:26.116277 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.116235    5098 generic.go:356] "Generic (PLEG): container finished" podID="8aa803d2-44a1-4de3-a49d-b751455fa261" containerID="9284f84ee7c207713cfe7bd9dd48ef95b00bb25495834e13904f95a36a8c3e33" exitCode=0
Oct 31 03:56:26.116473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.116316    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps" event={"ID":"8aa803d2-44a1-4de3-a49d-b751455fa261","Type":"ContainerDied","Data":"9284f84ee7c207713cfe7bd9dd48ef95b00bb25495834e13904f95a36a8c3e33"}
Oct 31 03:56:26.118289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.118264    5098 generic.go:356] "Generic (PLEG): container finished" podID="5d1e7197-521d-4bf2-9786-eb9a37a84b7d" containerID="9dbcaf19006ca0e65ca00c1823ab65a4dfa0bbb4d0c1ad4be1b6f0ecc198745d" exitCode=0
Oct 31 03:56:26.118402 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.118350    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp" event={"ID":"5d1e7197-521d-4bf2-9786-eb9a37a84b7d","Type":"ContainerDied","Data":"9dbcaf19006ca0e65ca00c1823ab65a4dfa0bbb4d0c1ad4be1b6f0ecc198745d"}
Oct 31 03:56:26.386762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.386692    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:26.437447 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.437406    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\") pod \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") "
Oct 31 03:56:26.437645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.437487    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\") pod \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") "
Oct 31 03:56:26.437645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.437592    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\" (UniqueName: \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\") pod \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\" (UID: \"470e0e9e-6fff-4aae-a82f-323f0a14bcbc\") "
Oct 31 03:56:26.440908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.440596    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln" pod "470e0e9e-6fff-4aae-a82f-323f0a14bcbc" (UID: "470e0e9e-6fff-4aae-a82f-323f0a14bcbc"). InnerVolumeSpecName "kube-api-access-97pln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:26.440908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.440864    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca" pod "470e0e9e-6fff-4aae-a82f-323f0a14bcbc" (UID: "470e0e9e-6fff-4aae-a82f-323f0a14bcbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:26.488210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.488172    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca" pod "470e0e9e-6fff-4aae-a82f-323f0a14bcbc" (UID: "470e0e9e-6fff-4aae-a82f-323f0a14bcbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:26.520636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.520581    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:26.533675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.533643    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:26.539492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.539459    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97pln\" (UniqueName: \"kubernetes.io/projected/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-kube-api-access-97pln\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:26.539492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.539485    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:26.539492 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.539496    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/470e0e9e-6fff-4aae-a82f-323f0a14bcbc-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:26.743943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.743851    5098 patch_prober.go:29] interesting pod/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:56:26.743943 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:26.743928    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:56:27.130465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.130436    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"
Oct 31 03:56:27.130648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.130466    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg" event={"ID":"470e0e9e-6fff-4aae-a82f-323f0a14bcbc","Type":"ContainerDied","Data":"8f4987eb7d1a4c4ff73daf49089869105a2222eaad8a350739fc09e42aa1c9b5"}
Oct 31 03:56:27.130648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.130518    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f4987eb7d1a4c4ff73daf49089869105a2222eaad8a350739fc09e42aa1c9b5"
Oct 31 03:56:27.146594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.146563    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:56:27.384723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.384640    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:27.388562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.388533    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:27.461291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.461248    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\") pod \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") "
Oct 31 03:56:27.461517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.461319    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\") pod \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") "
Oct 31 03:56:27.461517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.461361    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\" (UniqueName: \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\") pod \"8aa803d2-44a1-4de3-a49d-b751455fa261\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") "
Oct 31 03:56:27.461823 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.461445    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\") pod \"8aa803d2-44a1-4de3-a49d-b751455fa261\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") "
Oct 31 03:56:27.461997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.461977    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\") pod \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\" (UID: \"5d1e7197-521d-4bf2-9786-eb9a37a84b7d\") "
Oct 31 03:56:27.462493 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.462449    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca" pod "5d1e7197-521d-4bf2-9786-eb9a37a84b7d" (UID: "5d1e7197-521d-4bf2-9786-eb9a37a84b7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:27.462879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.462858    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\") pod \"8aa803d2-44a1-4de3-a49d-b751455fa261\" (UID: \"8aa803d2-44a1-4de3-a49d-b751455fa261\") "
Oct 31 03:56:27.463472 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.463352    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca" pod "8aa803d2-44a1-4de3-a49d-b751455fa261" (UID: "8aa803d2-44a1-4de3-a49d-b751455fa261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:27.464166 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464139    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494" pod "8aa803d2-44a1-4de3-a49d-b751455fa261" (UID: "8aa803d2-44a1-4de3-a49d-b751455fa261"). InnerVolumeSpecName "kube-api-access-sr494". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:27.464275 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464236    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj" pod "5d1e7197-521d-4bf2-9786-eb9a37a84b7d" (UID: "5d1e7197-521d-4bf2-9786-eb9a37a84b7d"). InnerVolumeSpecName "kube-api-access-nt8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:27.464440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464407    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca" pod "8aa803d2-44a1-4de3-a49d-b751455fa261" (UID: "8aa803d2-44a1-4de3-a49d-b751455fa261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:27.464917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464898    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt8mj\" (UniqueName: \"kubernetes.io/projected/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-kube-api-access-nt8mj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.464972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464924    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr494\" (UniqueName: \"kubernetes.io/projected/8aa803d2-44a1-4de3-a49d-b751455fa261-kube-api-access-sr494\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.464972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464939    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.464972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464956    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.464972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.464970    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aa803d2-44a1-4de3-a49d-b751455fa261-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.465116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.465067    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca" pod "5d1e7197-521d-4bf2-9786-eb9a37a84b7d" (UID: "5d1e7197-521d-4bf2-9786-eb9a37a84b7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:27.567028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.566983    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1e7197-521d-4bf2-9786-eb9a37a84b7d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:27.883432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:27.883401    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:56:28.134378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.134287    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:56:28.150252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.150063    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"
Oct 31 03:56:28.151892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.150848    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps" event={"ID":"8aa803d2-44a1-4de3-a49d-b751455fa261","Type":"ContainerDied","Data":"13a7974153fb635a4a419ee2a301b2402fb8c4d718a04fdc62593bcd7ad6a071"}
Oct 31 03:56:28.151892 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.150936    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13a7974153fb635a4a419ee2a301b2402fb8c4d718a04fdc62593bcd7ad6a071"
Oct 31 03:56:28.157383 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.157353    5098 generic.go:356] "Generic (PLEG): container finished" podID="944b1f85-9632-424d-b388-454fea87b870" containerID="687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701" exitCode=0
Oct 31 03:56:28.157522 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.157494    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerDied","Data":"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"}
Oct 31 03:56:28.157654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.157543    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6" event={"ID":"944b1f85-9632-424d-b388-454fea87b870","Type":"ContainerDied","Data":"c484468807173230a0e1d10c0adc27bb6489d8eadb73293518610a3d7857a39d"}
Oct 31 03:56:28.157654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.157577    5098 scope.go:122] "RemoveContainer" containerID="687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"
Oct 31 03:56:28.157784 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.157766    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"
Oct 31 03:56:28.162950 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.162899    5098 generic.go:356] "Generic (PLEG): container finished" podID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerID="7352a298d00977dff2344efa58357f229435e807df2e5c90355882ba52253506" exitCode=0
Oct 31 03:56:28.163229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.163004    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" event={"ID":"4114df17-feac-45ef-b7d7-42d39aaa39fd","Type":"ContainerDied","Data":"7352a298d00977dff2344efa58357f229435e807df2e5c90355882ba52253506"}
Oct 31 03:56:28.166183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.166142    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp" event={"ID":"5d1e7197-521d-4bf2-9786-eb9a37a84b7d","Type":"ContainerDied","Data":"5bac5b026d338228c7f00fb8a90848207aa096f2e200826e1b80ee88dec5b5fc"}
Oct 31 03:56:28.166325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.166199    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bac5b026d338228c7f00fb8a90848207aa096f2e200826e1b80ee88dec5b5fc"
Oct 31 03:56:28.166325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.166279    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"
Oct 31 03:56:28.171259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.169864    5098 generic.go:356] "Generic (PLEG): container finished" podID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerID="579f511660a1fd83d64101cff99bfcafd3a1266fc6765bcaadafd2e442771351" exitCode=0
Oct 31 03:56:28.171259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.170925    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" event={"ID":"ab131ed9-5f98-4b29-98ea-88a4f42137a8","Type":"ContainerDied","Data":"579f511660a1fd83d64101cff99bfcafd3a1266fc6765bcaadafd2e442771351"}
Oct 31 03:56:28.178763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.178724    5098 scope.go:122] "RemoveContainer" containerID="eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"
Oct 31 03:56:28.179933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.179905    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.180977    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181077    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181126    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181215    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181324    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181410    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181462    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.181548 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.181532    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\") pod \"944b1f85-9632-424d-b388-454fea87b870\" (UID: \"944b1f85-9632-424d-b388-454fea87b870\") "
Oct 31 03:56:28.182714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.182190    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:28.182714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.182472    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.182714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.182644    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.182714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.182665    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/944b1f85-9632-424d-b388-454fea87b870-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.183139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.183078    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.183218 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.183201    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.183286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.183254    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.186560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.186515    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:28.201363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.201002    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.219579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.217947    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "kube-api-access-bzsz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.219579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.218815    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token" pod "944b1f85-9632-424d-b388-454fea87b870" (UID: "944b1f85-9632-424d-b388-454fea87b870"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.284661 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.283684    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301706    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301745    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301764    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301780    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/944b1f85-9632-424d-b388-454fea87b870-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301801    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzsz6\" (UniqueName: \"kubernetes.io/projected/944b1f85-9632-424d-b388-454fea87b870-kube-api-access-bzsz6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301816    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.304837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.301829    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/944b1f85-9632-424d-b388-454fea87b870-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.305975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.305565    5098 scope.go:122] "RemoveContainer" containerID="687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"
Oct 31 03:56:28.306535 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:28.306235    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701\": container with ID starting with 687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701 not found: ID does not exist" containerID="687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"
Oct 31 03:56:28.306535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.306275    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701"} err="failed to get container status \"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701\": rpc error: code = NotFound desc = could not find container \"687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701\": container with ID starting with 687db8657ca90db24938c65c440703bd69b2a35fe85a32f34059c36b68ad1701 not found: ID does not exist"
Oct 31 03:56:28.306535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.306299    5098 scope.go:122] "RemoveContainer" containerID="eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"
Oct 31 03:56:28.308378 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:28.306846    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0\": container with ID starting with eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0 not found: ID does not exist" containerID="eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"
Oct 31 03:56:28.308378 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.306878    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0"} err="failed to get container status \"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0\": rpc error: code = NotFound desc = could not find container \"eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0\": container with ID starting with eebfda79b5a14c3405fa1b24305b32f31fbd8e37daa0cbe85db1fbe901ed14b0 not found: ID does not exist"
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408077    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408167    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408249    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408270    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408380    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408400    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408435    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408469    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.412949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.408530    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\") pod \"4114df17-feac-45ef-b7d7-42d39aaa39fd\" (UID: \"4114df17-feac-45ef-b7d7-42d39aaa39fd\") "
Oct 31 03:56:28.417518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.416273    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.417518 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.417465    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.418714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.418141    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.418714 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.418210    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.418889 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.418763    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:28.428240 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.428165    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.432480 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.432440    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.432862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.432812    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:28.433286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.433167    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w" pod "4114df17-feac-45ef-b7d7-42d39aaa39fd" (UID: "4114df17-feac-45ef-b7d7-42d39aaa39fd"). InnerVolumeSpecName "kube-api-access-7gx2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.502724 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.502688    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:56:28.506028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.506003    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-7xmt6"]
Oct 31 03:56:28.510890 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.510861    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511057 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511039    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511063    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511080    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511093    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511106    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511120    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4114df17-feac-45ef-b7d7-42d39aaa39fd-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511135    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gx2w\" (UniqueName: \"kubernetes.io/projected/4114df17-feac-45ef-b7d7-42d39aaa39fd-kube-api-access-7gx2w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.511414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.511151    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4114df17-feac-45ef-b7d7-42d39aaa39fd-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.634667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.634602    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:56:28.714646 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.714552    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.714844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.714655    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.714844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.714790    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.714844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.714822    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.714992    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715036 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715008    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.715138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715039    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715085    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715138 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715111    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.715302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715167    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715230    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.715302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715270    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\") pod \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\" (UID: \"ab131ed9-5f98-4b29-98ea-88a4f42137a8\") "
Oct 31 03:56:28.715569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715365    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.715569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.715396    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:28.716747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.716723    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.716941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.716915    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.716941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.716942    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.717110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.716955    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.717110 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.716968    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.718403 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.718380    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.718502 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.718453    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:28.718887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.718867    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "kube-api-access-n5c64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:28.718987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.718926    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo" pod "ab131ed9-5f98-4b29-98ea-88a4f42137a8" (UID: "ab131ed9-5f98-4b29-98ea-88a4f42137a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:28.825310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.824871    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.825310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.824911    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5c64\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-kube-api-access-n5c64\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.825310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.824926    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.825310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.824940    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab131ed9-5f98-4b29-98ea-88a4f42137a8-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:28.998012 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:28.997911    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:56:29.181145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.181112    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"
Oct 31 03:56:29.181360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.181137    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t" event={"ID":"4114df17-feac-45ef-b7d7-42d39aaa39fd","Type":"ContainerDied","Data":"cc7a639ae5486f8afd47bc4f3b87bdc0e2387de29086ae1b02b55bf4a7a8f1c7"}
Oct 31 03:56:29.181360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.181199    5098 scope.go:122] "RemoveContainer" containerID="7352a298d00977dff2344efa58357f229435e807df2e5c90355882ba52253506"
Oct 31 03:56:29.182997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.182965    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz" event={"ID":"ab131ed9-5f98-4b29-98ea-88a4f42137a8","Type":"ContainerDied","Data":"c1bd528ab365ab15537f8e7aace92017be3156011a87d5a5d632b6d605da8ca1"}
Oct 31 03:56:29.182997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.182989    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"
Oct 31 03:56:29.194258 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.194236    5098 scope.go:122] "RemoveContainer" containerID="579f511660a1fd83d64101cff99bfcafd3a1266fc6765bcaadafd2e442771351"
Oct 31 03:56:29.216193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.216152    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:56:29.223310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.223279    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-8m76t"]
Oct 31 03:56:29.241276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.241240    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:56:29.247652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.247587    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-afb8be-openshift-default-59dcd7f7bf-v7skz"]
Oct 31 03:56:29.613649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.613532    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" path="/var/lib/kubelet/pods/4114df17-feac-45ef-b7d7-42d39aaa39fd/volumes"
Oct 31 03:56:29.614248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.614222    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="944b1f85-9632-424d-b388-454fea87b870" path="/var/lib/kubelet/pods/944b1f85-9632-424d-b388-454fea87b870/volumes"
Oct 31 03:56:29.614878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.614856    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" path="/var/lib/kubelet/pods/ab131ed9-5f98-4b29-98ea-88a4f42137a8/volumes"
Oct 31 03:56:29.681014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.680979    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"]
Oct 31 03:56:29.681680 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681664    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5d1e7197-521d-4bf2-9786-eb9a37a84b7d" containerName="revoke-keys"
Oct 31 03:56:29.681728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681681    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1e7197-521d-4bf2-9786-eb9a37a84b7d" containerName="revoke-keys"
Oct 31 03:56:29.681728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681698    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy"
Oct 31 03:56:29.681728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681704    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy"
Oct 31 03:56:29.681728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681718    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4d5ab9f0-1aea-422f-a24e-8ae51101dd99" containerName="revoke-keys"
Oct 31 03:56:29.681728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681722    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5ab9f0-1aea-422f-a24e-8ae51101dd99" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681731    5098 cpu_manager.go:414] "containerMap: removing container" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681736    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681755    5098 cpu_manager.go:414] "containerMap: removing container" podUID="470e0e9e-6fff-4aae-a82f-323f0a14bcbc" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681760    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="470e0e9e-6fff-4aae-a82f-323f0a14bcbc" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681774    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681778    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681783    5098 cpu_manager.go:414] "containerMap: removing container" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681788    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681795    5098 cpu_manager.go:414] "containerMap: removing container" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681800    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681806    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d0e0ffc4-1d10-4a97-a95b-bd24c2150798" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681812    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e0ffc4-1d10-4a97-a95b-bd24c2150798" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681821    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681826    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681835    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8aa803d2-44a1-4de3-a49d-b751455fa261" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681840    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aa803d2-44a1-4de3-a49d-b751455fa261" containerName="revoke-keys"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681848    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681853    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681860    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy"
Oct 31 03:56:29.681903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681864    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681970    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aa803d2-44a1-4de3-a49d-b751455fa261" containerName="revoke-keys"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681977    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f404570-36f6-458e-b0cd-b64a3ae53f89" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681985    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4114df17-feac-45ef-b7d7-42d39aaa39fd" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681992    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1e7197-521d-4bf2-9786-eb9a37a84b7d" containerName="revoke-keys"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.681998    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab131ed9-5f98-4b29-98ea-88a4f42137a8" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682003    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5ab9f0-1aea-422f-a24e-8ae51101dd99" containerName="revoke-keys"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682010    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682017    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="470e0e9e-6fff-4aae-a82f-323f0a14bcbc" containerName="revoke-keys"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682024    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="944b1f85-9632-424d-b388-454fea87b870" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682030    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682037    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d7936d4-6ad3-4216-990d-d7cc5d26902c" containerName="istio-proxy"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682042    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e0ffc4-1d10-4a97-a95b-bd24c2150798" containerName="revoke-keys"
Oct 31 03:56:29.682499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.682048    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ea2218c-5720-4f94-af70-b8c0f5445a9b" containerName="istio-proxy"
Oct 31 03:56:29.700634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.700587    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"]
Oct 31 03:56:29.700864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.700723    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.706987 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.706963    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.739225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.739196    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"]
Oct 31 03:56:29.741929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.741899    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.742150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.742124    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp9js\" (UniqueName: \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.742259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.742243    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.742318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.742278    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzkll\" (UniqueName: \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.742318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.742297    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.742419 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.742330    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.743711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.743681    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:29.744139 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.744120    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:56:29.747601 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.747580    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"]
Oct 31 03:56:29.787079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.787042    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"]
Oct 31 03:56:29.794965 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.794930    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.806150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.806109    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"]
Oct 31 03:56:29.811407 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.811377    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:29.815375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.815349    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"]
Oct 31 03:56:29.827991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.827955    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"]
Oct 31 03:56:29.844091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844056    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.844260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844104    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp9js\" (UniqueName: \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.844260 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844203    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.844347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844264    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.844846 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844816    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.844949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844835    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzkll\" (UniqueName: \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.844949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844892    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.844949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844925    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g52v9\" (UniqueName: \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.845124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.844977    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.845124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.845058    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.846077 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.846051    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.847448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.847415    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.848026 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.847994    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.860293 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.860261    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp9js\" (UniqueName: \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\") pod \"maas-api-revoke-sub-27598604-fx9b5\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:29.867143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.867071    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzkll\" (UniqueName: \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\") pod \"maas-api-revoke-sub-6ebf6a69-5b9s7\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:29.946163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946123    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:29.946356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946265    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqsnt\" (UniqueName: \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:29.946356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946325    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:29.946500 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946366    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.946556 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946536    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g52v9\" (UniqueName: \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.946701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.946665    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.947302 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.947283    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.948900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.948879    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:29.967099 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:29.967064    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g52v9\" (UniqueName: \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\") pod \"maas-api-revoke-sub-535506ee-rfbbr\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:30.018568 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.018530    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:30.025209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.025183    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:30.047582 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.047547    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqsnt\" (UniqueName: \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.047742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.047626    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.047864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.047844    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.048521 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.048486    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.050543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.050515    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.089287 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.089251    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqsnt\" (UniqueName: \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\") pod \"maas-api-revoke-sub-29f66008-tm7xj\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.129880 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.129791    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:30.141587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.141561    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:30.369792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.369703    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"]
Oct 31 03:56:30.376744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.376674    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"]
Oct 31 03:56:30.439654 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.439597    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:56:30.511543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.511487    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"]
Oct 31 03:56:30.528818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:30.528758    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"]
Oct 31 03:56:30.554904 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:30.554832    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae038ac_e8a2_4c0a_bf9a_4a65e2aaf53d.slice/crio-f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c WatchSource:0}: Error finding container f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c: Status 404 returned error can't find the container with id f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c
Oct 31 03:56:30.555373 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:30.555344    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63589c56_1abc_4ce6_945e_5198907c324d.slice/crio-46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43 WatchSource:0}: Error finding container 46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43: Status 404 returned error can't find the container with id 46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43
Oct 31 03:56:31.061162 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:31.061079    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:56:31.276972 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.276927    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5" event={"ID":"1a7b6634-6adb-4a6e-95d1-747a689247d2","Type":"ContainerStarted","Data":"1e5bb85a2f5788936eb535efeed77122c2e4235c0210d748ef618fa6174a0999"}
Oct 31 03:56:31.277161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.277039    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5" event={"ID":"1a7b6634-6adb-4a6e-95d1-747a689247d2","Type":"ContainerStarted","Data":"4c058828aeac2977c1d0119af62aa0fe133143ddb75fcb9651ef92d4b6410e71"}
Oct 31 03:56:31.279161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.279125    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj" event={"ID":"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d","Type":"ContainerStarted","Data":"564eda6acbec082e5c8d87bcc0fdc1472405a1db275933946b30db21a1b8154c"}
Oct 31 03:56:31.279286 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.279179    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj" event={"ID":"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d","Type":"ContainerStarted","Data":"f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c"}
Oct 31 03:56:31.280970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.280943    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr" event={"ID":"63589c56-1abc-4ce6-945e-5198907c324d","Type":"ContainerStarted","Data":"7ac1f1fad601d7fe09ba3e781b89db9794de6b871abc979e5bb2dd08781eb513"}
Oct 31 03:56:31.281094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.280984    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr" event={"ID":"63589c56-1abc-4ce6-945e-5198907c324d","Type":"ContainerStarted","Data":"46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43"}
Oct 31 03:56:31.282805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.282777    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7" event={"ID":"6c9e1dfc-f753-463e-ac40-060d52a1607b","Type":"ContainerStarted","Data":"c59f4fb5c2b8716bf1bed170d562203144b40c8ef55bf4f79da84204268a7f79"}
Oct 31 03:56:31.282968 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.282829    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7" event={"ID":"6c9e1dfc-f753-463e-ac40-060d52a1607b","Type":"ContainerStarted","Data":"20cb877f045c7c63b92bd4ce306575c3f1087920df6f24bef1b7dba0816f75d0"}
Oct 31 03:56:31.300686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.300579    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5" podStartSLOduration=2.30056312 podStartE2EDuration="2.30056312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:31.298720812 +0000 UTC m=+2465.823940406" watchObservedRunningTime="2026-10-31 03:56:31.30056312 +0000 UTC m=+2465.825782708"
Oct 31 03:56:31.323852 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.323793    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr" podStartSLOduration=2.323777633 podStartE2EDuration="2.323777633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:31.318505593 +0000 UTC m=+2465.843725183" watchObservedRunningTime="2026-10-31 03:56:31.323777633 +0000 UTC m=+2465.848997242"
Oct 31 03:56:31.350000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.349917    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj" podStartSLOduration=2.349892847 podStartE2EDuration="2.349892847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:31.345993041 +0000 UTC m=+2465.871212630" watchObservedRunningTime="2026-10-31 03:56:31.349892847 +0000 UTC m=+2465.875112437"
Oct 31 03:56:31.372474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:31.372417    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7" podStartSLOduration=2.372401242 podStartE2EDuration="2.372401242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:31.369670091 +0000 UTC m=+2465.894889681" watchObservedRunningTime="2026-10-31 03:56:31.372401242 +0000 UTC m=+2465.897620856"
Oct 31 03:56:32.129560 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.129517    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:56:32.138667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.136064    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.144956 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.144925    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-39e8cf\"/\"default-dockercfg-plzlg\""
Oct 31 03:56:32.145433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.145404    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:56:32.146641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.146596    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-39e8cf\"/\"kube-root-ca.crt\""
Oct 31 03:56:32.146773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.146673    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-39e8cf\"/\"openshift-service-ca.crt\""
Oct 31 03:56:32.146773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.146700    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-39e8cf\"/\"auth-test-model-39e8cf-kserve-self-signed-certs\""
Oct 31 03:56:32.188897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.188858    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.188897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.188906    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.189196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.189085    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98hf8\" (UniqueName: \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.189196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.189136    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.189338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.189227    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.189338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.189264    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326034    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98hf8\" (UniqueName: \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326194    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326247    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326298    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.326549    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.333633 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.327029    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.345185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.345153    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.355015 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.354984    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.360513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.359948    5098 generic.go:356] "Generic (PLEG): container finished" podID="63589c56-1abc-4ce6-945e-5198907c324d" containerID="7ac1f1fad601d7fe09ba3e781b89db9794de6b871abc979e5bb2dd08781eb513" exitCode=0
Oct 31 03:56:32.360513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.360074    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98hf8\" (UniqueName: \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.360513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.360152    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr" event={"ID":"63589c56-1abc-4ce6-945e-5198907c324d","Type":"ContainerDied","Data":"7ac1f1fad601d7fe09ba3e781b89db9794de6b871abc979e5bb2dd08781eb513"}
Oct 31 03:56:32.362719 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.362684    5098 generic.go:356] "Generic (PLEG): container finished" podID="6c9e1dfc-f753-463e-ac40-060d52a1607b" containerID="c59f4fb5c2b8716bf1bed170d562203144b40c8ef55bf4f79da84204268a7f79" exitCode=0
Oct 31 03:56:32.363276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.363248    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7" event={"ID":"6c9e1dfc-f753-463e-ac40-060d52a1607b","Type":"ContainerDied","Data":"c59f4fb5c2b8716bf1bed170d562203144b40c8ef55bf4f79da84204268a7f79"}
Oct 31 03:56:32.369841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.367508    5098 generic.go:356] "Generic (PLEG): container finished" podID="1a7b6634-6adb-4a6e-95d1-747a689247d2" containerID="1e5bb85a2f5788936eb535efeed77122c2e4235c0210d748ef618fa6174a0999" exitCode=0
Oct 31 03:56:32.369841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.367674    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5" event={"ID":"1a7b6634-6adb-4a6e-95d1-747a689247d2","Type":"ContainerDied","Data":"1e5bb85a2f5788936eb535efeed77122c2e4235c0210d748ef618fa6174a0999"}
Oct 31 03:56:32.370292 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.370261    5098 generic.go:356] "Generic (PLEG): container finished" podID="dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" containerID="564eda6acbec082e5c8d87bcc0fdc1472405a1db275933946b30db21a1b8154c" exitCode=0
Oct 31 03:56:32.370423 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.370305    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj" event={"ID":"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d","Type":"ContainerDied","Data":"564eda6acbec082e5c8d87bcc0fdc1472405a1db275933946b30db21a1b8154c"}
Oct 31 03:56:32.392262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.391125    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.392262 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.391907    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\") pod \"auth-test-model-39e8cf-kserve-65d599f47c-xzfk8\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") " pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.464499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.464462    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:32.861945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.861570    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:56:32.890752 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:32.888370    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" podUID="d03172c7-d030-405c-a795-5313a22c806d" containerName="payload-processing" containerID="cri-o://4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b" gracePeriod=25
Oct 31 03:56:32.935769 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:32.935167    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae2082e8_d563_408f_a911_12646033a523.slice/crio-49eb1996b410413968d54f6da08964296068631c058ca23f181088318d9ff4c9 WatchSource:0}: Error finding container 49eb1996b410413968d54f6da08964296068631c058ca23f181088318d9ff4c9: Status 404 returned error can't find the container with id 49eb1996b410413968d54f6da08964296068631c058ca23f181088318d9ff4c9
Oct 31 03:56:33.005499 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:33.005330    5098 logging.go:55] [core] [Channel #8496 SubChannel #8497]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.64:50052", ServerName: "10.132.0.64:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.64:50052: connect: connection refused"
Oct 31 03:56:33.226490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.226457    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:33.294357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.294325    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:56:33.354078 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.353980    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.354257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354132    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.354326 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354299    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.354515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354427    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:33.354702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354490    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:33.354775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354758    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:33.354875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354854    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.354938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354917    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.355008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.354992    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.355155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.355130    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:33.355217 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.355160    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.355274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.355218    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.355274 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.355263    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\") pod \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\" (UID: \"1aaad8d5-3920-44c9-baa9-a0e1674f5e08\") "
Oct 31 03:56:33.356041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.355978    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.356041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.356011    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.356041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.356026    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.356041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.356042    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.356513 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.356486    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:33.357844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.357815    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:33.358270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.358242    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:33.358494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.358470    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:33.358849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.358823    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h" pod "1aaad8d5-3920-44c9-baa9-a0e1674f5e08" (UID: "1aaad8d5-3920-44c9-baa9-a0e1674f5e08"). InnerVolumeSpecName "kube-api-access-jpz7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:33.385307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.385265    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" event={"ID":"ae2082e8-d563-408f-a911-12646033a523","Type":"ContainerStarted","Data":"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"}
Oct 31 03:56:33.385482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.385357    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" event={"ID":"ae2082e8-d563-408f-a911-12646033a523","Type":"ContainerStarted","Data":"49eb1996b410413968d54f6da08964296068631c058ca23f181088318d9ff4c9"}
Oct 31 03:56:33.387543 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.387517    5098 generic.go:356] "Generic (PLEG): container finished" podID="d03172c7-d030-405c-a795-5313a22c806d" containerID="4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b" exitCode=0
Oct 31 03:56:33.387686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.387577    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"
Oct 31 03:56:33.387686 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.387570    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" event={"ID":"d03172c7-d030-405c-a795-5313a22c806d","Type":"ContainerDied","Data":"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"}
Oct 31 03:56:33.387811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.387711    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" event={"ID":"d03172c7-d030-405c-a795-5313a22c806d","Type":"ContainerDied","Data":"66268a9b7a41ba440f86bf5e8b841b5461ca84937ecb951873849d60631dbb7a"}
Oct 31 03:56:33.387811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.387761    5098 scope.go:122] "RemoveContainer" containerID="4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"
Oct 31 03:56:33.389800 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.389774    5098 generic.go:356] "Generic (PLEG): container finished" podID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerID="9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674" exitCode=0
Oct 31 03:56:33.389884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.389872    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"
Oct 31 03:56:33.389926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.389898    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" event={"ID":"1aaad8d5-3920-44c9-baa9-a0e1674f5e08","Type":"ContainerDied","Data":"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"}
Oct 31 03:56:33.389970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.389949    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7" event={"ID":"1aaad8d5-3920-44c9-baa9-a0e1674f5e08","Type":"ContainerDied","Data":"05ac0c21cf0cb0aaa0ee6e426b2e8c4beb8f94ae341a6eedb012bdeb8613eec6"}
Oct 31 03:56:33.403641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.403591    5098 scope.go:122] "RemoveContainer" containerID="4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"
Oct 31 03:56:33.404328 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:33.404284    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b\": container with ID starting with 4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b not found: ID does not exist" containerID="4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"
Oct 31 03:56:33.404424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.404342    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b"} err="failed to get container status \"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b\": rpc error: code = NotFound desc = could not find container \"4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b\": container with ID starting with 4578884f4bea2049680ba4bcd193d55b5e2b495edeba25da056001150414ad8b not found: ID does not exist"
Oct 31 03:56:33.404424 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.404369    5098 scope.go:122] "RemoveContainer" containerID="9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"
Oct 31 03:56:33.423702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.420680    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" podStartSLOduration=1.420659793 podStartE2EDuration="1.420659793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:33.41950391 +0000 UTC m=+2467.944723500" watchObservedRunningTime="2026-10-31 03:56:33.420659793 +0000 UTC m=+2467.945879384"
Oct 31 03:56:33.431295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.431259    5098 scope.go:122] "RemoveContainer" containerID="9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"
Oct 31 03:56:33.431867 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:33.431758    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674\": container with ID starting with 9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674 not found: ID does not exist" containerID="9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"
Oct 31 03:56:33.431867 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.431800    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674"} err="failed to get container status \"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674\": rpc error: code = NotFound desc = could not find container \"9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674\": container with ID starting with 9af85d3e1f4a97f8b4e73f6a596b4da9f3b25bc7693f54af0313ef9c8c4ec674 not found: ID does not exist"
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.463635    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\" (UniqueName: \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\") pod \"d03172c7-d030-405c-a795-5313a22c806d\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") "
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.463731    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\") pod \"d03172c7-d030-405c-a795-5313a22c806d\" (UID: \"d03172c7-d030-405c-a795-5313a22c806d\") "
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.464599    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.464639    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.464655    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpz7h\" (UniqueName: \"kubernetes.io/projected/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-kube-api-access-jpz7h\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.464669    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.466444 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.464684    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1aaad8d5-3920-44c9-baa9-a0e1674f5e08-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.468038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.467965    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:33.469091 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.468985    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume" pod "d03172c7-d030-405c-a795-5313a22c806d" (UID: "d03172c7-d030-405c-a795-5313a22c806d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:33.473632 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.473559    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:33.475376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.475321    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42" pod "d03172c7-d030-405c-a795-5313a22c806d" (UID: "d03172c7-d030-405c-a795-5313a22c806d"). InnerVolumeSpecName "kube-api-access-mgk42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:33.495870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.485487    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ce9b89-openshift-default-6c64956cd9-q5pt7"]
Oct 31 03:56:33.566489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.566454    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgk42\" (UniqueName: \"kubernetes.io/projected/d03172c7-d030-405c-a795-5313a22c806d-kube-api-access-mgk42\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.566489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.566486    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d03172c7-d030-405c-a795-5313a22c806d-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:33.619364 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.619283    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" path="/var/lib/kubelet/pods/1aaad8d5-3920-44c9-baa9-a0e1674f5e08/volumes"
Oct 31 03:56:33.760635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.754822    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:33.903696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.903654    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:56:33.913128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.913096    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg"]
Oct 31 03:56:33.913365 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.913218    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:33.968650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.968291    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\" (UniqueName: \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\") pod \"1a7b6634-6adb-4a6e-95d1-747a689247d2\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") "
Oct 31 03:56:33.968650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.968343    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\") pod \"1a7b6634-6adb-4a6e-95d1-747a689247d2\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") "
Oct 31 03:56:33.968650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.968414    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\") pod \"1a7b6634-6adb-4a6e-95d1-747a689247d2\" (UID: \"1a7b6634-6adb-4a6e-95d1-747a689247d2\") "
Oct 31 03:56:33.968650 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.968556    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\") pod \"63589c56-1abc-4ce6-945e-5198907c324d\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") "
Oct 31 03:56:33.972136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.969089    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\") pod \"63589c56-1abc-4ce6-945e-5198907c324d\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") "
Oct 31 03:56:33.972136 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.969160    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\" (UniqueName: \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\") pod \"63589c56-1abc-4ce6-945e-5198907c324d\" (UID: \"63589c56-1abc-4ce6-945e-5198907c324d\") "
Oct 31 03:56:33.977495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.976830    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:33.981992 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.981735    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9" pod "63589c56-1abc-4ce6-945e-5198907c324d" (UID: "63589c56-1abc-4ce6-945e-5198907c324d"). InnerVolumeSpecName "kube-api-access-g52v9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:33.982313 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.982282    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca" pod "63589c56-1abc-4ce6-945e-5198907c324d" (UID: "63589c56-1abc-4ce6-945e-5198907c324d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:33.982858 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.982764    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca" pod "63589c56-1abc-4ce6-945e-5198907c324d" (UID: "63589c56-1abc-4ce6-945e-5198907c324d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:33.983014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.982985    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca" pod "1a7b6634-6adb-4a6e-95d1-747a689247d2" (UID: "1a7b6634-6adb-4a6e-95d1-747a689247d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:33.983371 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.983310    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca" pod "1a7b6634-6adb-4a6e-95d1-747a689247d2" (UID: "1a7b6634-6adb-4a6e-95d1-747a689247d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:33.983828 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.983795    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js" pod "1a7b6634-6adb-4a6e-95d1-747a689247d2" (UID: "1a7b6634-6adb-4a6e-95d1-747a689247d2"). InnerVolumeSpecName "kube-api-access-jp9js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:33.985131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.985046    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:33.999143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:33.999072    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing" containerID="cri-o://8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723" gracePeriod=25
Oct 31 03:56:34.005473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.005401    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-dda5cb-59bbb9d57-5nrbg" podUID="d03172c7-d030-405c-a795-5313a22c806d" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.64:50052\" within 1s: context deadline exceeded"
Oct 31 03:56:34.071308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071270    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\") pod \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") "
Oct 31 03:56:34.071547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071327    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\") pod \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") "
Oct 31 03:56:34.071547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071401    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\" (UniqueName: \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\") pod \"6c9e1dfc-f753-463e-ac40-060d52a1607b\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") "
Oct 31 03:56:34.071547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071461    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\") pod \"6c9e1dfc-f753-463e-ac40-060d52a1607b\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") "
Oct 31 03:56:34.071547 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071516    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\" (UniqueName: \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\") pod \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\" (UID: \"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d\") "
Oct 31 03:56:34.071793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071764    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\") pod \"6c9e1dfc-f753-463e-ac40-060d52a1607b\" (UID: \"6c9e1dfc-f753-463e-ac40-060d52a1607b\") "
Oct 31 03:56:34.071864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071810    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca" pod "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" (UID: "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.071927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.071888    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca" pod "6c9e1dfc-f753-463e-ac40-060d52a1607b" (UID: "6c9e1dfc-f753-463e-ac40-060d52a1607b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:34.072458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072427    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g52v9\" (UniqueName: \"kubernetes.io/projected/63589c56-1abc-4ce6-945e-5198907c324d-kube-api-access-g52v9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072460    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072477    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp9js\" (UniqueName: \"kubernetes.io/projected/1a7b6634-6adb-4a6e-95d1-747a689247d2-kube-api-access-jp9js\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072492    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072510    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072526    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a7b6634-6adb-4a6e-95d1-747a689247d2-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072541    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.072595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.072556    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63589c56-1abc-4ce6-945e-5198907c324d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.073975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.073945    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca" pod "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" (UID: "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:34.073975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.073942    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt" pod "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" (UID: "dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d"). InnerVolumeSpecName "kube-api-access-gqsnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.074123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.073953    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll" pod "6c9e1dfc-f753-463e-ac40-060d52a1607b" (UID: "6c9e1dfc-f753-463e-ac40-060d52a1607b"). InnerVolumeSpecName "kube-api-access-kzkll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.074123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.074029    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca" pod "6c9e1dfc-f753-463e-ac40-060d52a1607b" (UID: "6c9e1dfc-f753-463e-ac40-060d52a1607b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:34.087857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.087818    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main" containerID="cri-o://0ac9d4f2852554fc58ccbff26ace61753036f373c77387e92fd90388d7b1899a" gracePeriod=45
Oct 31 03:56:34.174267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.174167    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzkll\" (UniqueName: \"kubernetes.io/projected/6c9e1dfc-f753-463e-ac40-060d52a1607b-kube-api-access-kzkll\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.174267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.174206    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqsnt\" (UniqueName: \"kubernetes.io/projected/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-kube-api-access-gqsnt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.174267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.174218    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c9e1dfc-f753-463e-ac40-060d52a1607b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.174267 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.174228    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.412154 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.412119    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"
Oct 31 03:56:34.412592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.412119    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7" event={"ID":"6c9e1dfc-f753-463e-ac40-060d52a1607b","Type":"ContainerDied","Data":"20cb877f045c7c63b92bd4ce306575c3f1087920df6f24bef1b7dba0816f75d0"}
Oct 31 03:56:34.412592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.412317    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20cb877f045c7c63b92bd4ce306575c3f1087920df6f24bef1b7dba0816f75d0"
Oct 31 03:56:34.415458 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.415430    5098 generic.go:356] "Generic (PLEG): container finished" podID="70ba18cb-cbc9-46e0-af51-873aba870133" containerID="0ac9d4f2852554fc58ccbff26ace61753036f373c77387e92fd90388d7b1899a" exitCode=0
Oct 31 03:56:34.415595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.415506    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" event={"ID":"70ba18cb-cbc9-46e0-af51-873aba870133","Type":"ContainerDied","Data":"0ac9d4f2852554fc58ccbff26ace61753036f373c77387e92fd90388d7b1899a"}
Oct 31 03:56:34.415595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.415565    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw" event={"ID":"70ba18cb-cbc9-46e0-af51-873aba870133","Type":"ContainerDied","Data":"95f7456853352e9dc8b5fcc90708907e10308445040d9a2463acd03f311f470c"}
Oct 31 03:56:34.415595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.415582    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95f7456853352e9dc8b5fcc90708907e10308445040d9a2463acd03f311f470c"
Oct 31 03:56:34.417577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.417552    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"
Oct 31 03:56:34.418246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.418217    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5" event={"ID":"1a7b6634-6adb-4a6e-95d1-747a689247d2","Type":"ContainerDied","Data":"4c058828aeac2977c1d0119af62aa0fe133143ddb75fcb9651ef92d4b6410e71"}
Oct 31 03:56:34.418375 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.418273    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c058828aeac2977c1d0119af62aa0fe133143ddb75fcb9651ef92d4b6410e71"
Oct 31 03:56:34.420535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.420234    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj" event={"ID":"dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d","Type":"ContainerDied","Data":"f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c"}
Oct 31 03:56:34.420535 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.420288    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f26e6e9b161b30ac54a1da6d37e0696fadb01fcad8acd6c31ef6704dace7d44c"
Oct 31 03:56:34.420947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.420927    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"
Oct 31 03:56:34.422255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.422235    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"
Oct 31 03:56:34.422381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.422277    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr" event={"ID":"63589c56-1abc-4ce6-945e-5198907c324d","Type":"ContainerDied","Data":"46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43"}
Oct 31 03:56:34.422381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.422317    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46044a4f189fbfeed0166dd1726a609c67238eec759206ca823288488e177c43"
Oct 31 03:56:34.449841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.449805    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:56:34.486102 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.486046    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.486566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.486539    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.486908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.486887    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.487140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.487116    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\" (UniqueName: \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.487261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.487178    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\" (UniqueName: \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.487261 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.487251    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\") pod \"70ba18cb-cbc9-46e0-af51-873aba870133\" (UID: \"70ba18cb-cbc9-46e0-af51-873aba870133\") "
Oct 31 03:56:34.491807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.491100    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.491807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.491306    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.491807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.491753    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.492344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.492318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:34.494617 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.494577    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:34.494974 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.494951    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q" pod "70ba18cb-cbc9-46e0-af51-873aba870133" (UID: "70ba18cb-cbc9-46e0-af51-873aba870133"). InnerVolumeSpecName "kube-api-access-bfn6q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:34.589125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589082    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.589125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589120    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ba18cb-cbc9-46e0-af51-873aba870133-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.589125 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589133    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfn6q\" (UniqueName: \"kubernetes.io/projected/70ba18cb-cbc9-46e0-af51-873aba870133-kube-api-access-bfn6q\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.589381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589145    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.589381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589153    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:34.589381 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:34.589161    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ba18cb-cbc9-46e0-af51-873aba870133-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:35.259459 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.259355    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:56:35.402193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.402160    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"]
Oct 31 03:56:35.402969 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.402947    5098 cpu_manager.go:414] "containerMap: removing container" podUID="dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.402972    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403001    5098 cpu_manager.go:414] "containerMap: removing container" podUID="63589c56-1abc-4ce6-945e-5198907c324d" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403007    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63589c56-1abc-4ce6-945e-5198907c324d" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403018    5098 cpu_manager.go:414] "containerMap: removing container" podUID="6c9e1dfc-f753-463e-ac40-060d52a1607b" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403024    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c9e1dfc-f753-463e-ac40-060d52a1607b" containerName="revoke-keys"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403033    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403037    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403048    5098 cpu_manager.go:414] "containerMap: removing container" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main"
Oct 31 03:56:35.403058 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403053    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403069    5098 cpu_manager.go:414] "containerMap: removing container" podUID="1a7b6634-6adb-4a6e-95d1-747a689247d2" containerName="revoke-keys"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403074    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a7b6634-6adb-4a6e-95d1-747a689247d2" containerName="revoke-keys"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403083    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d03172c7-d030-405c-a795-5313a22c806d" containerName="payload-processing"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403088    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03172c7-d030-405c-a795-5313a22c806d" containerName="payload-processing"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403178    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="63589c56-1abc-4ce6-945e-5198907c324d" containerName="revoke-keys"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403187    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c9e1dfc-f753-463e-ac40-060d52a1607b" containerName="revoke-keys"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403194    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aaad8d5-3920-44c9-baa9-a0e1674f5e08" containerName="istio-proxy"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403200    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" containerName="main"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403209    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a7b6634-6adb-4a6e-95d1-747a689247d2" containerName="revoke-keys"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403216    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03172c7-d030-405c-a795-5313a22c806d" containerName="payload-processing"
Oct 31 03:56:35.403406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.403223    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" containerName="revoke-keys"
Oct 31 03:56:35.407317 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.407297    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.412853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.412818    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:56:35.413346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.413323    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:35.418862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.417120    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"]
Oct 31 03:56:35.423437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.423412    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.424245 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.424220    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"]
Oct 31 03:56:35.439995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.437684    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"
Oct 31 03:56:35.439995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.438790    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"]
Oct 31 03:56:35.444084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.444011    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing" containerID="cri-o://49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598" gracePeriod=25
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562105    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562269    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczml\" (UniqueName: \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562393    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562421    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562539    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.612857 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.562639    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4dd\" (UniqueName: \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.665234 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665191    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.665465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665296    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4dd\" (UniqueName: \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.665465 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665408    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.665593 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665529    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczml\" (UniqueName: \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.665674 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665646    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.665738 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.665673    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.667037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.667010    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.668766 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.668427    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.676653 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.675160    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.678648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.676830    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03172c7-d030-405c-a795-5313a22c806d" path="/var/lib/kubelet/pods/d03172c7-d030-405c-a795-5313a22c806d/volumes"
Oct 31 03:56:35.678648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.677533    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"]
Oct 31 03:56:35.678648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.677569    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-fc5ece-xx4kp"]
Oct 31 03:56:35.678648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.677594    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:56:35.678648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.677634    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d86a77/simulator-w0-d86a77-kserve-5b788df88b-rfplw"]
Oct 31 03:56:35.679014 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.678939    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4dd\" (UniqueName: \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.679112 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.679088    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0aad177f-pcsm4\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.679448 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.679428    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczml\" (UniqueName: \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\") pod \"maas-api-revoke-sub-da5422ce-ltmj9\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:35.735731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.735685    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:35.953426 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:35.953388    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:36.035135 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:36.035101    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"]
Oct 31 03:56:36.249066 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:36.249031    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4fce454_2352_4351_89ee_219b65ba27aa.slice/crio-4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e WatchSource:0}: Error finding container 4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e: Status 404 returned error can't find the container with id 4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e
Oct 31 03:56:36.251301 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:36.251270    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"]
Oct 31 03:56:36.449951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:36.449910    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9" event={"ID":"f4fce454-2352-4351-89ee-219b65ba27aa","Type":"ContainerStarted","Data":"4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e"}
Oct 31 03:56:36.451759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:36.451720    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4" event={"ID":"40d94e96-6a61-4a9e-9a6d-f5526a07a957","Type":"ContainerStarted","Data":"ada27ce13d52ff795c39818651c9a006366954efc7e98655d9782e7746a8536e"}
Oct 31 03:56:37.474508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.474403    5098 generic.go:356] "Generic (PLEG): container finished" podID="f4fce454-2352-4351-89ee-219b65ba27aa" containerID="60d269a48a4db22514a784a36afac3f9d23770d9cc527112104a49b2b8dcc49a" exitCode=0
Oct 31 03:56:37.474508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.474490    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9" event={"ID":"f4fce454-2352-4351-89ee-219b65ba27aa","Type":"ContainerDied","Data":"60d269a48a4db22514a784a36afac3f9d23770d9cc527112104a49b2b8dcc49a"}
Oct 31 03:56:37.476783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.476752    5098 generic.go:356] "Generic (PLEG): container finished" podID="40d94e96-6a61-4a9e-9a6d-f5526a07a957" containerID="9eca71df12fdec2b95b3a11cb4a56d504fba90a8042fc3ea086d3a9657541ba8" exitCode=0
Oct 31 03:56:37.476945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.476837    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4" event={"ID":"40d94e96-6a61-4a9e-9a6d-f5526a07a957","Type":"ContainerDied","Data":"9eca71df12fdec2b95b3a11cb4a56d504fba90a8042fc3ea086d3a9657541ba8"}
Oct 31 03:56:37.615524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.615487    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1e7197-521d-4bf2-9786-eb9a37a84b7d" path="/var/lib/kubelet/pods/5d1e7197-521d-4bf2-9786-eb9a37a84b7d/volumes"
Oct 31 03:56:37.616011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.615991    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ba18cb-cbc9-46e0-af51-873aba870133" path="/var/lib/kubelet/pods/70ba18cb-cbc9-46e0-af51-873aba870133/volumes"
Oct 31 03:56:37.808648 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.808597    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"]
Oct 31 03:56:37.813159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:37.813120    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-86a40d-5gdpl"]
Oct 31 03:56:38.715311 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.715284    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:38.739177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.739148    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:38.747951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.747867    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\") pod \"f4fce454-2352-4351-89ee-219b65ba27aa\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") "
Oct 31 03:56:38.747951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.747925    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\") pod \"f4fce454-2352-4351-89ee-219b65ba27aa\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") "
Oct 31 03:56:38.748194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.748057    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\") pod \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") "
Oct 31 03:56:38.748785 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.748756    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\" (UniqueName: \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\") pod \"f4fce454-2352-4351-89ee-219b65ba27aa\" (UID: \"f4fce454-2352-4351-89ee-219b65ba27aa\") "
Oct 31 03:56:38.748925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.748847    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca" pod "f4fce454-2352-4351-89ee-219b65ba27aa" (UID: "f4fce454-2352-4351-89ee-219b65ba27aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:38.749241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.749217    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\") pod \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") "
Oct 31 03:56:38.749833 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.749714    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca" pod "40d94e96-6a61-4a9e-9a6d-f5526a07a957" (UID: "40d94e96-6a61-4a9e-9a6d-f5526a07a957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:38.750041 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.750021    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\" (UniqueName: \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\") pod \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\" (UID: \"40d94e96-6a61-4a9e-9a6d-f5526a07a957\") "
Oct 31 03:56:38.750998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.750973    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.751126 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.751112    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.754340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.754310    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml" pod "f4fce454-2352-4351-89ee-219b65ba27aa" (UID: "f4fce454-2352-4351-89ee-219b65ba27aa"). InnerVolumeSpecName "kube-api-access-cczml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:38.754476 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.754416    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca" pod "f4fce454-2352-4351-89ee-219b65ba27aa" (UID: "f4fce454-2352-4351-89ee-219b65ba27aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:38.759353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.755100    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca" pod "40d94e96-6a61-4a9e-9a6d-f5526a07a957" (UID: "40d94e96-6a61-4a9e-9a6d-f5526a07a957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:38.759353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.757958    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd" pod "40d94e96-6a61-4a9e-9a6d-f5526a07a957" (UID: "40d94e96-6a61-4a9e-9a6d-f5526a07a957"). InnerVolumeSpecName "kube-api-access-gp4dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:38.853579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.852584    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40d94e96-6a61-4a9e-9a6d-f5526a07a957-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.853579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.852635    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cczml\" (UniqueName: \"kubernetes.io/projected/f4fce454-2352-4351-89ee-219b65ba27aa-kube-api-access-cczml\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.853579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.852648    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp4dd\" (UniqueName: \"kubernetes.io/projected/40d94e96-6a61-4a9e-9a6d-f5526a07a957-kube-api-access-gp4dd\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:38.853579 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:38.852657    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4fce454-2352-4351-89ee-219b65ba27aa-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:39.503811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.503776    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"
Oct 31 03:56:39.503994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.503810    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9" event={"ID":"f4fce454-2352-4351-89ee-219b65ba27aa","Type":"ContainerDied","Data":"4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e"}
Oct 31 03:56:39.503994 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.503871    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c4a1ac12e0b7dd99927ed946cfdb3b682ec8a93ca4a8c9d7f45f4b2b76a521e"
Oct 31 03:56:39.505474 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.505450    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"
Oct 31 03:56:39.505677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.505475    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4" event={"ID":"40d94e96-6a61-4a9e-9a6d-f5526a07a957","Type":"ContainerDied","Data":"ada27ce13d52ff795c39818651c9a006366954efc7e98655d9782e7746a8536e"}
Oct 31 03:56:39.505677 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.505519    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada27ce13d52ff795c39818651c9a006366954efc7e98655d9782e7746a8536e"
Oct 31 03:56:39.615580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.614649    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e0ffc4-1d10-4a97-a95b-bd24c2150798" path="/var/lib/kubelet/pods/d0e0ffc4-1d10-4a97-a95b-bd24c2150798/volumes"
Oct 31 03:56:39.964437 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.964403    5098 scope.go:122] "RemoveContainer" containerID="97130b945d7adb42185eb5540d5024cbcfca57d29d485e24336d426811db80c4"
Oct 31 03:56:39.980835 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.980808    5098 scope.go:122] "RemoveContainer" containerID="70021509e2951e6abf0d111af4f8bd6e5e70d27f44dc54a6eb048fbfaf791124"
Oct 31 03:56:39.992566 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:39.992541    5098 scope.go:122] "RemoveContainer" containerID="0ac9d4f2852554fc58ccbff26ace61753036f373c77387e92fd90388d7b1899a"
Oct 31 03:56:40.003082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.003059    5098 scope.go:122] "RemoveContainer" containerID="9475ac2f430449f2451875371243959990e2f55571b53789257d0e6f66ae34c9"
Oct 31 03:56:40.033257 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.033219    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" containerID="cri-o://f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba" gracePeriod=45
Oct 31 03:56:40.194871 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.194836    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.96:8000/ready\": dial tcp 10.132.0.96:8000: connect: connection refused"
Oct 31 03:56:40.370816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.370780    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:56:40.476796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.476706    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\" (UniqueName: \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.476796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.476781    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.477043 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.476894    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\" (UniqueName: \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.477109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477092    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.477222 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477203    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.477307 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477237    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\") pod \"2342cb51-d06e-44b6-bd12-bc80eedeed06\" (UID: \"2342cb51-d06e-44b6-bd12-bc80eedeed06\") "
Oct 31 03:56:40.477414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477390    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:40.477494 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477439    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:40.477744 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.477713    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:40.478053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.478025    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:40.478185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.478057    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:40.478185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.478071    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:40.480040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.480008    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:40.480040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.480026    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "kube-api-access-nv9gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:40.480229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.480034    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs" pod "2342cb51-d06e-44b6-bd12-bc80eedeed06" (UID: "2342cb51-d06e-44b6-bd12-bc80eedeed06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:40.516118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.516080    5098 generic.go:356] "Generic (PLEG): container finished" podID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerID="f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba" exitCode=0
Oct 31 03:56:40.516282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.516131    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"
Oct 31 03:56:40.516282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.516165    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" event={"ID":"2342cb51-d06e-44b6-bd12-bc80eedeed06","Type":"ContainerDied","Data":"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"}
Oct 31 03:56:40.516282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.516215    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk" event={"ID":"2342cb51-d06e-44b6-bd12-bc80eedeed06","Type":"ContainerDied","Data":"a47b4269872a53710d4badf763f8c25e3882832f708b7d2fb0a15778ef74fd60"}
Oct 31 03:56:40.516282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.516254    5098 scope.go:122] "RemoveContainer" containerID="f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"
Oct 31 03:56:40.528360 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.528328    5098 scope.go:122] "RemoveContainer" containerID="f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"
Oct 31 03:56:40.528706 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:40.528683    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba\": container with ID starting with f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba not found: ID does not exist" containerID="f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"
Oct 31 03:56:40.528795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.528716    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba"} err="failed to get container status \"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba\": rpc error: code = NotFound desc = could not find container \"f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba\": container with ID starting with f33037771778efec9894ed68cffbcc42a333b717950d19e82e485ecabadc2eba not found: ID does not exist"
Oct 31 03:56:40.545079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.545049    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:56:40.549370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.549345    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-6c6050/distinct-w1-6c6050-kserve-7bf55d86b4-fdvmk"]
Oct 31 03:56:40.608600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.608011    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv9gf\" (UniqueName: \"kubernetes.io/projected/2342cb51-d06e-44b6-bd12-bc80eedeed06-kube-api-access-nv9gf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:40.608600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.608063    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2342cb51-d06e-44b6-bd12-bc80eedeed06-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:40.608600 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:40.608085    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2342cb51-d06e-44b6-bd12-bc80eedeed06-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:41.613163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:41.613122    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" path="/var/lib/kubelet/pods/2342cb51-d06e-44b6-bd12-bc80eedeed06/volumes"
Oct 31 03:56:42.465177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:42.465139    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:42.478782 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:42.478751    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:42.555269 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:42.555236    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:56:42.896452 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:42.896346    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"]
Oct 31 03:56:42.905595 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:42.905566    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d04aad-zwgps"]
Oct 31 03:56:43.540962 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.540914    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"]
Oct 31 03:56:43.541938 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.541919    5098 cpu_manager.go:414] "containerMap: removing container" podUID="40d94e96-6a61-4a9e-9a6d-f5526a07a957" containerName="revoke-keys"
Oct 31 03:56:43.542018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.541940    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d94e96-6a61-4a9e-9a6d-f5526a07a957" containerName="revoke-keys"
Oct 31 03:56:43.542018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.541956    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f4fce454-2352-4351-89ee-219b65ba27aa" containerName="revoke-keys"
Oct 31 03:56:43.542018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.541964    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4fce454-2352-4351-89ee-219b65ba27aa" containerName="revoke-keys"
Oct 31 03:56:43.542018 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.542016    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main"
Oct 31 03:56:43.542177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.542024    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main"
Oct 31 03:56:43.542177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.542158    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d94e96-6a61-4a9e-9a6d-f5526a07a957" containerName="revoke-keys"
Oct 31 03:56:43.542177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.542170    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4fce454-2352-4351-89ee-219b65ba27aa" containerName="revoke-keys"
Oct 31 03:56:43.542271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.542185    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2342cb51-d06e-44b6-bd12-bc80eedeed06" containerName="main"
Oct 31 03:56:43.547882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.547852    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.554721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.554684    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"]
Oct 31 03:56:43.595425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.595384    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:43.595657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.595432    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:56:43.613945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.613908    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aa803d2-44a1-4de3-a49d-b751455fa261" path="/var/lib/kubelet/pods/8aa803d2-44a1-4de3-a49d-b751455fa261/volumes"
Oct 31 03:56:43.691081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.691035    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.691291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.691123    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm578\" (UniqueName: \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.691596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.691574    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.793149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.793043    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.793351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.793201    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.793351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.793254    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm578\" (UniqueName: \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.793933 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.793908    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.795743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.795715    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.810238 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.810206    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm578\" (UniqueName: \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:43.869562 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:43.869522    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:44.014098 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:44.014053    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:56:44.269751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:44.269712    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"]
Oct 31 03:56:44.291430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:44.291397    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"]
Oct 31 03:56:44.300391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:44.296229    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-dda5cb-rr69n"]
Oct 31 03:56:44.558409 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:44.558370    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm" event={"ID":"421c2537-cf77-4db8-a3e5-6a2fdb375c40","Type":"ContainerStarted","Data":"fb6ff75a37bf3cdf1a5bf1021797372382067a23a8447953b04ab85030963e65"}
Oct 31 03:56:45.258729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.258691    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:56:45.360066 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.360030    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"]
Oct 31 03:56:45.364787 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.364752    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-d86a77-q22pv"]
Oct 31 03:56:45.586887 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.586849    5098 generic.go:356] "Generic (PLEG): container finished" podID="421c2537-cf77-4db8-a3e5-6a2fdb375c40" containerID="b90cd3a631e56b3cbaf53584b0b87d7ecb1f4b8bda3355a99a4c87ed813d7a7f" exitCode=0
Oct 31 03:56:45.587233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.587203    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm" event={"ID":"421c2537-cf77-4db8-a3e5-6a2fdb375c40","Type":"ContainerDied","Data":"b90cd3a631e56b3cbaf53584b0b87d7ecb1f4b8bda3355a99a4c87ed813d7a7f"}
Oct 31 03:56:45.613429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.613395    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5ab9f0-1aea-422f-a24e-8ae51101dd99" path="/var/lib/kubelet/pods/4d5ab9f0-1aea-422f-a24e-8ae51101dd99/volumes"
Oct 31 03:56:45.614053 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:45.614035    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd061516-8162-4c3e-b970-e0b967ca65b8" path="/var/lib/kubelet/pods/cd061516-8162-4c3e-b970-e0b967ca65b8/volumes"
Oct 31 03:56:46.501107 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.501073    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:46.501651 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.501328    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy" containerID="cri-o://c2e7e53e170aeb3e6b02e36c4cf59ab5a44faa59e944e28ded976826ec1e6182" gracePeriod=30
Oct 31 03:56:46.899309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.899274    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:46.972570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.972528    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\") pod \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") "
Oct 31 03:56:46.972793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.972673    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\") pod \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") "
Oct 31 03:56:46.972793 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.972776    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\" (UniqueName: \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\") pod \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\" (UID: \"421c2537-cf77-4db8-a3e5-6a2fdb375c40\") "
Oct 31 03:56:46.973096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.973071    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca" pod "421c2537-cf77-4db8-a3e5-6a2fdb375c40" (UID: "421c2537-cf77-4db8-a3e5-6a2fdb375c40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:46.973499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.973477    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:46.974822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.974795    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578" pod "421c2537-cf77-4db8-a3e5-6a2fdb375c40" (UID: "421c2537-cf77-4db8-a3e5-6a2fdb375c40"). InnerVolumeSpecName "kube-api-access-mm578". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:46.974932 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:46.974844    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca" pod "421c2537-cf77-4db8-a3e5-6a2fdb375c40" (UID: "421c2537-cf77-4db8-a3e5-6a2fdb375c40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:47.074598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:47.074510    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421c2537-cf77-4db8-a3e5-6a2fdb375c40-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:47.074598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:47.074544    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm578\" (UniqueName: \"kubernetes.io/projected/421c2537-cf77-4db8-a3e5-6a2fdb375c40-kube-api-access-mm578\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:47.674168 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:47.674133    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"
Oct 31 03:56:47.683874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:47.683839    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm" event={"ID":"421c2537-cf77-4db8-a3e5-6a2fdb375c40","Type":"ContainerDied","Data":"fb6ff75a37bf3cdf1a5bf1021797372382067a23a8447953b04ab85030963e65"}
Oct 31 03:56:47.684068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:47.683893    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb6ff75a37bf3cdf1a5bf1021797372382067a23a8447953b04ab85030963e65"
Oct 31 03:56:51.473841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:51.473803    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:56:51.474344 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:51.474008    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" containerID="cri-o://5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860" gracePeriod=30
Oct 31 03:56:51.722688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:51.722651    5098 generic.go:356] "Generic (PLEG): container finished" podID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerID="c2e7e53e170aeb3e6b02e36c4cf59ab5a44faa59e944e28ded976826ec1e6182" exitCode=0
Oct 31 03:56:51.722886 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:51.722723    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" event={"ID":"c3ddb675-e819-4515-9606-ef5b0dcd60e0","Type":"ContainerDied","Data":"c2e7e53e170aeb3e6b02e36c4cf59ab5a44faa59e944e28ded976826ec1e6182"}
Oct 31 03:56:51.938780 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:51.938749    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:52.027164 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027129    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027209    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027233    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027288    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027396    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027486    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027517 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027402    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:52.027712 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027694    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027779 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027731    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027811    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\") pod \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\" (UID: \"c3ddb675-e819-4515-9606-ef5b0dcd60e0\") "
Oct 31 03:56:52.027837 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027820    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:52.027952 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.027846    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:52.028028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028005    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:52.028116 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028094    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:52.028429 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028409    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.028523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028433    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.028523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028448    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.028523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028462    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.028523 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.028475    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.030163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.030140    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "kube-api-access-k2scf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:52.030481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.030457    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:52.030807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.030787    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:52.030879 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.030862    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo" pod "c3ddb675-e819-4515-9606-ef5b0dcd60e0" (UID: "c3ddb675-e819-4515-9606-ef5b0dcd60e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:52.130418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.130324    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.130418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.130365    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.130418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.130376    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c3ddb675-e819-4515-9606-ef5b0dcd60e0-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.130418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.130385    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2scf\" (UniqueName: \"kubernetes.io/projected/c3ddb675-e819-4515-9606-ef5b0dcd60e0-kube-api-access-k2scf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:52.220809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.220770    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"]
Oct 31 03:56:52.221529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221508    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy"
Oct 31 03:56:52.221658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221531    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy"
Oct 31 03:56:52.221658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221561    5098 cpu_manager.go:414] "containerMap: removing container" podUID="421c2537-cf77-4db8-a3e5-6a2fdb375c40" containerName="revoke-keys"
Oct 31 03:56:52.221658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221571    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421c2537-cf77-4db8-a3e5-6a2fdb375c40" containerName="revoke-keys"
Oct 31 03:56:52.221840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221798    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" containerName="istio-proxy"
Oct 31 03:56:52.221840 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.221817    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="421c2537-cf77-4db8-a3e5-6a2fdb375c40" containerName="revoke-keys"
Oct 31 03:56:52.225951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.225926    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.231592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.231563    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.231844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.231696    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqlf\" (UniqueName: \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.231844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.231830    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.232182 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.232162    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:56:52.232255 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.232220    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:56:52.243055 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.243022    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"]
Oct 31 03:56:52.333213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.333170    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.333398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.333261    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqlf\" (UniqueName: \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.333398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.333330    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.334095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.334063    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.335703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.335676    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.341882 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.341857    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqlf\" (UniqueName: \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\") pod \"maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.544008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.543975    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:52.733482 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.733451    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"
Oct 31 03:56:52.733733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.733477    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp" event={"ID":"c3ddb675-e819-4515-9606-ef5b0dcd60e0","Type":"ContainerDied","Data":"ebcaef113aabb26804f362f3025fc00c61e1151493b16c2b4fafd86f3e73f32a"}
Oct 31 03:56:52.733733 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.733550    5098 scope.go:122] "RemoveContainer" containerID="c2e7e53e170aeb3e6b02e36c4cf59ab5a44faa59e944e28ded976826ec1e6182"
Oct 31 03:56:52.765694 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.765653    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:52.769897 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.769869    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-86a40d-openshift-default-98b679fc5-4cbmp"]
Oct 31 03:56:52.832351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:52.832310    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"]
Oct 31 03:56:52.832859 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:56:52.832821    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode97769d0_2654_4e5a_9111_b127962ae022.slice/crio-ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a WatchSource:0}: Error finding container ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a: Status 404 returned error can't find the container with id ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a
Oct 31 03:56:53.513324 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.513228    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:56:53.513508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.513450    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy" containerID="cri-o://805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1" gracePeriod=30
Oct 31 03:56:53.612742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.612711    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3ddb675-e819-4515-9606-ef5b0dcd60e0" path="/var/lib/kubelet/pods/c3ddb675-e819-4515-9606-ef5b0dcd60e0/volumes"
Oct 31 03:56:53.784524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.783212    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:56:53.784524 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.783462    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" containerID="cri-o://c8e4538a3de963d6f988a359e9131ea28aad1c2423d040b3dbc84006c8da9051" gracePeriod=30
Oct 31 03:56:53.816572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.816527    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7" event={"ID":"e97769d0-2654-4e5a-9111-b127962ae022","Type":"ContainerStarted","Data":"4f5c67234304a0f7f4fce39b16969de1d11e57c14d2d471d07149ba1db509538"}
Oct 31 03:56:53.817216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.817134    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7" event={"ID":"e97769d0-2654-4e5a-9111-b127962ae022","Type":"ContainerStarted","Data":"ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a"}
Oct 31 03:56:53.883845 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:53.883785    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7" podStartSLOduration=1.8837657239999999 podStartE2EDuration="1.883765724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:56:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:56:53.877676185 +0000 UTC m=+2488.402895771" watchObservedRunningTime="2026-10-31 03:56:53.883765724 +0000 UTC m=+2488.408985361"
Oct 31 03:56:54.056671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.050822    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:56:54.383051 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.383018    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:56:54.394197 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.394165    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\" (UniqueName: \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\") pod \"7d0ca89e-3ab1-42cc-a513-f76920562b57\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") "
Oct 31 03:56:54.394347 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.394256    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\") pod \"7d0ca89e-3ab1-42cc-a513-f76920562b57\" (UID: \"7d0ca89e-3ab1-42cc-a513-f76920562b57\") "
Oct 31 03:56:54.394620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.394580    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume" pod "7d0ca89e-3ab1-42cc-a513-f76920562b57" (UID: "7d0ca89e-3ab1-42cc-a513-f76920562b57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:54.394736 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.394720    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d0ca89e-3ab1-42cc-a513-f76920562b57-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:54.396380 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.396361    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf" pod "7d0ca89e-3ab1-42cc-a513-f76920562b57" (UID: "7d0ca89e-3ab1-42cc-a513-f76920562b57"). InnerVolumeSpecName "kube-api-access-vzhhf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:54.495848 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.495808    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzhhf\" (UniqueName: \"kubernetes.io/projected/7d0ca89e-3ab1-42cc-a513-f76920562b57-kube-api-access-vzhhf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:54.917630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.909925    5098 generic.go:356] "Generic (PLEG): container finished" podID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerID="8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723" exitCode=0
Oct 31 03:56:54.917630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.910056    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" event={"ID":"7d0ca89e-3ab1-42cc-a513-f76920562b57","Type":"ContainerDied","Data":"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"}
Oct 31 03:56:54.917630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.913509    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"
Oct 31 03:56:54.917630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.915037    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm" event={"ID":"7d0ca89e-3ab1-42cc-a513-f76920562b57","Type":"ContainerDied","Data":"e44b5fb5e80cbb0d8ca7c6ea3ae68e31ab0073bf3b8bd5981bf5f0c56c484262"}
Oct 31 03:56:54.917630 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.915097    5098 scope.go:122] "RemoveContainer" containerID="8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"
Oct 31 03:56:54.925180 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.925134    5098 generic.go:356] "Generic (PLEG): container finished" podID="e97769d0-2654-4e5a-9111-b127962ae022" containerID="4f5c67234304a0f7f4fce39b16969de1d11e57c14d2d471d07149ba1db509538" exitCode=0
Oct 31 03:56:54.925446 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.925423    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7" event={"ID":"e97769d0-2654-4e5a-9111-b127962ae022","Type":"ContainerDied","Data":"4f5c67234304a0f7f4fce39b16969de1d11e57c14d2d471d07149ba1db509538"}
Oct 31 03:56:54.964899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.954773    5098 scope.go:122] "RemoveContainer" containerID="8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"
Oct 31 03:56:54.964899 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:54.960845    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723\": container with ID starting with 8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723 not found: ID does not exist" containerID="8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"
Oct 31 03:56:54.964899 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:54.960906    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723"} err="failed to get container status \"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723\": rpc error: code = NotFound desc = could not find container \"8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723\": container with ID starting with 8029bef0ac4886b18566fa8e3f54ac2db64fa789cf23a2f2c277158c5e4f2723 not found: ID does not exist"
Oct 31 03:56:55.009173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.009112    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"]
Oct 31 03:56:55.016348 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.016312    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-6c6050-r6mqg"]
Oct 31 03:56:55.023629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.023568    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:56:55.028049 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.028014    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-86a40d-6d78d9f86c-bltwm"]
Oct 31 03:56:55.615069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.615031    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="470e0e9e-6fff-4aae-a82f-323f0a14bcbc" path="/var/lib/kubelet/pods/470e0e9e-6fff-4aae-a82f-323f0a14bcbc/volumes"
Oct 31 03:56:55.615588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.615564    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" path="/var/lib/kubelet/pods/7d0ca89e-3ab1-42cc-a513-f76920562b57/volumes"
Oct 31 03:56:55.751358 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.751329    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:56:55.826995 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.826934    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\") pod \"9a0374ab-5f65-4545-ba95-bda2f7a56913\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") "
Oct 31 03:56:55.827241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.827147    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\" (UniqueName: \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\") pod \"9a0374ab-5f65-4545-ba95-bda2f7a56913\" (UID: \"9a0374ab-5f65-4545-ba95-bda2f7a56913\") "
Oct 31 03:56:55.828645 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.828593    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume" pod "9a0374ab-5f65-4545-ba95-bda2f7a56913" (UID: "9a0374ab-5f65-4545-ba95-bda2f7a56913"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:55.890636 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.890164    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k" pod "9a0374ab-5f65-4545-ba95-bda2f7a56913" (UID: "9a0374ab-5f65-4545-ba95-bda2f7a56913"). InnerVolumeSpecName "kube-api-access-hls5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:55.918658 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.918568    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 03:56:55.919165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.918672    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 03:56:55.942631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.929723    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a0374ab-5f65-4545-ba95-bda2f7a56913-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:55.942631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.929764    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hls5k\" (UniqueName: \"kubernetes.io/projected/9a0374ab-5f65-4545-ba95-bda2f7a56913-kube-api-access-hls5k\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:55.957561 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.956240    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"]
Oct 31 03:56:55.961259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961218    5098 generic.go:356] "Generic (PLEG): container finished" podID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerID="49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598" exitCode=0
Oct 31 03:56:55.961635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961267    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"
Oct 31 03:56:55.961635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961285    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" event={"ID":"9a0374ab-5f65-4545-ba95-bda2f7a56913","Type":"ContainerDied","Data":"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"}
Oct 31 03:56:55.961635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961338    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4" event={"ID":"9a0374ab-5f65-4545-ba95-bda2f7a56913","Type":"ContainerDied","Data":"95f524d1d740c2fc4d3438c0098a2611a8f9a8866666d16bc967cbe58a6acb68"}
Oct 31 03:56:55.961635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961359    5098 scope.go:122] "RemoveContainer" containerID="49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"
Oct 31 03:56:55.962056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.961692    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5396c2-7lhsm"]
Oct 31 03:56:55.977945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.977921    5098 scope.go:122] "RemoveContainer" containerID="49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"
Oct 31 03:56:55.978274 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:55.978253    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598\": container with ID starting with 49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598 not found: ID does not exist" containerID="49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"
Oct 31 03:56:55.978333 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.978285    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598"} err="failed to get container status \"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598\": rpc error: code = NotFound desc = could not find container \"49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598\": container with ID starting with 49a554faea493c597cf89ba11ab73a2fe82eb40ecd261c5da0de66ca8483f598 not found: ID does not exist"
Oct 31 03:56:55.988350 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.988316    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:56:55.994945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:55.994913    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-6c6050-6d4c44c5c5-5r6m4"]
Oct 31 03:56:56.180028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.179998    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:56.348473 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.348428    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\") pod \"e97769d0-2654-4e5a-9111-b127962ae022\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") "
Oct 31 03:56:56.348715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.348487    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\" (UniqueName: \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\") pod \"e97769d0-2654-4e5a-9111-b127962ae022\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") "
Oct 31 03:56:56.348715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.348562    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\") pod \"e97769d0-2654-4e5a-9111-b127962ae022\" (UID: \"e97769d0-2654-4e5a-9111-b127962ae022\") "
Oct 31 03:56:56.349111 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.349000    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca" pod "e97769d0-2654-4e5a-9111-b127962ae022" (UID: "e97769d0-2654-4e5a-9111-b127962ae022"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:56.349643 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.349601    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.351083 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.351051    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf" pod "e97769d0-2654-4e5a-9111-b127962ae022" (UID: "e97769d0-2654-4e5a-9111-b127962ae022"). InnerVolumeSpecName "kube-api-access-vxqlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:56.351184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.351141    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca" pod "e97769d0-2654-4e5a-9111-b127962ae022" (UID: "e97769d0-2654-4e5a-9111-b127962ae022"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:56:56.450851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.450806    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e97769d0-2654-4e5a-9111-b127962ae022-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.450851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.450845    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqlf\" (UniqueName: \"kubernetes.io/projected/e97769d0-2654-4e5a-9111-b127962ae022-kube-api-access-vxqlf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.867699 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.867661    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:56:56.962751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962656    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.962751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962750    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962808    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962833    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962868    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.962908    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963000    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963089    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963138    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963217    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963226    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\") pod \"ce21f832-9004-4bea-aa30-01340b4c84a2\" (UID: \"ce21f832-9004-4bea-aa30-01340b4c84a2\") "
Oct 31 03:56:56.963284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963267    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:56:56.963841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963461    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:56.963841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963498    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:56.963951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963870    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.963951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963892    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.963951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963903    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce21f832-9004-4bea-aa30-01340b4c84a2-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.963951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.963912    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:56.964203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.964052    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:56.966549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.966517    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:56.966549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.966523    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:56.966806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.966542    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:56.967900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.967868    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l" pod "ce21f832-9004-4bea-aa30-01340b4c84a2" (UID: "ce21f832-9004-4bea-aa30-01340b4c84a2"). InnerVolumeSpecName "kube-api-access-bh65l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:56.975666 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.975635    5098 generic.go:356] "Generic (PLEG): container finished" podID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerID="5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860" exitCode=0
Oct 31 03:56:56.975816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.975702    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"
Oct 31 03:56:56.975816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.975758    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerDied","Data":"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"}
Oct 31 03:56:56.975816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.975806    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b" event={"ID":"ce21f832-9004-4bea-aa30-01340b4c84a2","Type":"ContainerDied","Data":"09ae89c33819a4800686d8387535bef8fd2115d8f78ab2a9b66320d21b8d7d49"}
Oct 31 03:56:56.975985 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.975832    5098 scope.go:122] "RemoveContainer" containerID="5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"
Oct 31 03:56:56.978687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.977843    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"
Oct 31 03:56:56.978687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.977871    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7" event={"ID":"e97769d0-2654-4e5a-9111-b127962ae022","Type":"ContainerDied","Data":"ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a"}
Oct 31 03:56:56.978687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.977917    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca47468514ec491886df079c4d1358f6bdb365c6da5e0324cb4cdda9adb4290a"
Oct 31 03:56:56.992527 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:56.992506    5098 scope.go:122] "RemoveContainer" containerID="e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"
Oct 31 03:56:57.003788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.003758    5098 scope.go:122] "RemoveContainer" containerID="5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"
Oct 31 03:56:57.004135 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:57.004114    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860\": container with ID starting with 5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860 not found: ID does not exist" containerID="5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"
Oct 31 03:56:57.004203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.004147    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860"} err="failed to get container status \"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860\": rpc error: code = NotFound desc = could not find container \"5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860\": container with ID starting with 5ad65ac4dca7a0b3a80d9330e73c08bd5a570b20422e589fc9e5147376858860 not found: ID does not exist"
Oct 31 03:56:57.004203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.004168    5098 scope.go:122] "RemoveContainer" containerID="e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"
Oct 31 03:56:57.004478 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:57.004461    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab\": container with ID starting with e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab not found: ID does not exist" containerID="e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"
Oct 31 03:56:57.004530 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.004483    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab"} err="failed to get container status \"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab\": rpc error: code = NotFound desc = could not find container \"e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab\": container with ID starting with e929eb8ba4c0bd470cdd6b8100e0f26448fee4ab41b3a244063aa046ad0103ab not found: ID does not exist"
Oct 31 03:56:57.010884 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.010855    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:56:57.015398 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.015373    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-d04aad-openshift-default-8676954f7f-f8v2b"]
Oct 31 03:56:57.065418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.065381    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:57.065418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.065412    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:57.065418 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.065427    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce21f832-9004-4bea-aa30-01340b4c84a2-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:57.065664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.065441    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh65l\" (UniqueName: \"kubernetes.io/projected/ce21f832-9004-4bea-aa30-01340b4c84a2-kube-api-access-bh65l\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:57.065664 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.065453    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce21f832-9004-4bea-aa30-01340b4c84a2-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:57.613349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.613309    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421c2537-cf77-4db8-a3e5-6a2fdb375c40" path="/var/lib/kubelet/pods/421c2537-cf77-4db8-a3e5-6a2fdb375c40/volumes"
Oct 31 03:56:57.613900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.613880    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" path="/var/lib/kubelet/pods/9a0374ab-5f65-4545-ba95-bda2f7a56913/volumes"
Oct 31 03:56:57.614346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:57.614332    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" path="/var/lib/kubelet/pods/ce21f832-9004-4bea-aa30-01340b4c84a2/volumes"
Oct 31 03:56:58.802629 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.802525    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:56:58.814775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814739    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.814940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814813    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.814940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814842    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.814940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814911    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814961    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814983    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:58.815085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.814998    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815078    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815179    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815199    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:58.815356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815228    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\") pod \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\" (UID: \"e74b0119-7795-4e12-9a3f-9e3824fa0df1\") "
Oct 31 03:56:58.815356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815249    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:58.815356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815266    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:58.815923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815884    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.815923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815894    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:58.815923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815910    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.815923 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815926    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.816169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.815939    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.817468 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.817445    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:58.817563 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.817546    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "kube-api-access-c9pnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:58.817874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.817841    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:58.817874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.817855    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy" pod "e74b0119-7795-4e12-9a3f-9e3824fa0df1" (UID: "e74b0119-7795-4e12-9a3f-9e3824fa0df1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:58.917149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.917109    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9pnd\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-kube-api-access-c9pnd\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.917149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.917144    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.917149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.917154    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.917149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.917163    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:58.917514 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:58.917172    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e74b0119-7795-4e12-9a3f-9e3824fa0df1-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.014590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.014550    5098 generic.go:356] "Generic (PLEG): container finished" podID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerID="c8e4538a3de963d6f988a359e9131ea28aad1c2423d040b3dbc84006c8da9051" exitCode=0
Oct 31 03:56:59.014822 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.014599    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" event={"ID":"f6c6bb59-428a-4433-9362-437e10c1c24f","Type":"ContainerDied","Data":"c8e4538a3de963d6f988a359e9131ea28aad1c2423d040b3dbc84006c8da9051"}
Oct 31 03:56:59.016374 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.016335    5098 generic.go:356] "Generic (PLEG): container finished" podID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerID="805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1" exitCode=0
Oct 31 03:56:59.016489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.016391    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"
Oct 31 03:56:59.016557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.016542    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" event={"ID":"e74b0119-7795-4e12-9a3f-9e3824fa0df1","Type":"ContainerDied","Data":"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"}
Oct 31 03:56:59.016635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.016582    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw" event={"ID":"e74b0119-7795-4e12-9a3f-9e3824fa0df1","Type":"ContainerDied","Data":"5fb3f1e208fa3369b21ac23a8b46d95e886f9ef4d00ff77b5a6a961db837fee1"}
Oct 31 03:56:59.016635 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.016620    5098 scope.go:122] "RemoveContainer" containerID="805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"
Oct 31 03:56:59.033799 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.033774    5098 scope.go:122] "RemoveContainer" containerID="805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"
Oct 31 03:56:59.034189 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:56:59.034162    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1\": container with ID starting with 805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1 not found: ID does not exist" containerID="805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"
Oct 31 03:56:59.034295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.034197    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1"} err="failed to get container status \"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1\": rpc error: code = NotFound desc = could not find container \"805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1\": container with ID starting with 805ab97820e782dc1840277a0bfb1c78890be1950ee4af0f2fe297c422f20ee1 not found: ID does not exist"
Oct 31 03:56:59.051114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.051078    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:56:59.057520 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.057445    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-dda5cb-openshift-default-5964d6d9ff-g8ntw"]
Oct 31 03:56:59.098035 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.098006    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:56:59.118622 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.118582    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.118874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.118713    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.118874 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.118768    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.119069 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.118996    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:56:59.119312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119208    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:59.119740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119472    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.119740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119522    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.119740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119594    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.119740 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119704    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.120008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119765    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.120008 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.119805    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\") pod \"f6c6bb59-428a-4433-9362-437e10c1c24f\" (UID: \"f6c6bb59-428a-4433-9362-437e10c1c24f\") "
Oct 31 03:56:59.120256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.120218    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:59.120256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.120249    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:59.120596 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.120546    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:59.121073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121038    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.121073 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121066    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.121235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121080    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f6c6bb59-428a-4433-9362-437e10c1c24f-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.121235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121094    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.121235 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121109    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.121905 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.121874    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:56:59.122816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.122791    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "kube-api-access-v97sr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:59.123004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.122984    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:56:59.123185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.123155    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo" pod "f6c6bb59-428a-4433-9362-437e10c1c24f" (UID: "f6c6bb59-428a-4433-9362-437e10c1c24f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:56:59.222246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.222204    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.222246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.222242    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.222246 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.222253    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.222587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.222263    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v97sr\" (UniqueName: \"kubernetes.io/projected/f6c6bb59-428a-4433-9362-437e10c1c24f-kube-api-access-v97sr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:56:59.616389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:56:59.616349    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" path="/var/lib/kubelet/pods/e74b0119-7795-4e12-9a3f-9e3824fa0df1/volumes"
Oct 31 03:57:00.027454 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.027432    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"
Oct 31 03:57:00.027893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.027430    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" event={"ID":"f6c6bb59-428a-4433-9362-437e10c1c24f","Type":"ContainerDied","Data":"fc8af709e78960b2a59725cb5f22a3900df0f411e345635940f097863c192199"}
Oct 31 03:57:00.027893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.027566    5098 scope.go:122] "RemoveContainer" containerID="c8e4538a3de963d6f988a359e9131ea28aad1c2423d040b3dbc84006c8da9051"
Oct 31 03:57:00.066023 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.065986    5098 patch_prober.go:29] interesting pod/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 03:57:00.066216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.066049    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 03:57:00.067143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.067119    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:57:00.083696 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:00.083656    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-d86a77-openshift-default-6d4dcf7499-wbn2q"]
Oct 31 03:57:01.058024 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:01.057987    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:57:01.612751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:01.612713    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" path="/var/lib/kubelet/pods/f6c6bb59-428a-4433-9362-437e10c1c24f/volumes"
Oct 31 03:57:02.367469 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:02.367423    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:57:03.396948 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:03.396847    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:57:03.966149 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:03.966114    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:57:03.966367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:03.966328    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy" containerID="cri-o://8e18f430341a50f6c639e5c3338507e39cea999cbd2481918611114d5fbfa740" gracePeriod=30
Oct 31 03:57:07.371741 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:07.371679    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing" containerID="cri-o://df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4" gracePeriod=25
Oct 31 03:57:09.121071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.120979    5098 generic.go:356] "Generic (PLEG): container finished" podID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerID="8e18f430341a50f6c639e5c3338507e39cea999cbd2481918611114d5fbfa740" exitCode=0
Oct 31 03:57:09.121071 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.121003    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" event={"ID":"b6b42b50-0b86-484f-9e70-f56ae576c4d0","Type":"ContainerDied","Data":"8e18f430341a50f6c639e5c3338507e39cea999cbd2481918611114d5fbfa740"}
Oct 31 03:57:09.189801 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.189764    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"]
Oct 31 03:57:09.190934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.190911    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing"
Oct 31 03:57:09.191088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191075    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing"
Oct 31 03:57:09.191194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191185    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e97769d0-2654-4e5a-9111-b127962ae022" containerName="revoke-keys"
Oct 31 03:57:09.191271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191262    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97769d0-2654-4e5a-9111-b127962ae022" containerName="revoke-keys"
Oct 31 03:57:09.191349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191341    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.191417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191409    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.191508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191500    5098 cpu_manager.go:414] "containerMap: removing container" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy"
Oct 31 03:57:09.191586 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191577    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy"
Oct 31 03:57:09.191762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191749    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.191849 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191840    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.191927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191919    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy"
Oct 31 03:57:09.192004 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.191995    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy"
Oct 31 03:57:09.192081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192073    5098 cpu_manager.go:414] "containerMap: removing container" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing"
Oct 31 03:57:09.192162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192152    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing"
Oct 31 03:57:09.192414 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192401    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e74b0119-7795-4e12-9a3f-9e3824fa0df1" containerName="istio-proxy"
Oct 31 03:57:09.192515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192506    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.192597 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192589    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6c6bb59-428a-4433-9362-437e10c1c24f" containerName="istio-proxy"
Oct 31 03:57:09.192700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192691    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d0ca89e-3ab1-42cc-a513-f76920562b57" containerName="payload-pre-processing"
Oct 31 03:57:09.192808 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192775    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0374ab-5f65-4545-ba95-bda2f7a56913" containerName="payload-pre-processing"
Oct 31 03:57:09.192895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192887    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97769d0-2654-4e5a-9111-b127962ae022" containerName="revoke-keys"
Oct 31 03:57:09.192971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.192963    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce21f832-9004-4bea-aa30-01340b4c84a2" containerName="istio-proxy"
Oct 31 03:57:09.202551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.202520    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.210598 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.209772    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"]
Oct 31 03:57:09.228778 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.228724    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:57:09.228980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.228784    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:57:09.315996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.315957    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.316214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.316028    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.316214 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.316193    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8pxb\" (UniqueName: \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.324739 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.324711    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:57:09.417310 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417270    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417322    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417369    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417444    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417495 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417479    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417535    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417583    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417759 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417688    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417759    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\") pod \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\" (UID: \"b6b42b50-0b86-484f-9e70-f56ae576c4d0\") "
Oct 31 03:57:09.417934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417772    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:09.417934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417880    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:09.418084 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.417927    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:09.418144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418090    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:09.418144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418111    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:09.418144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418131    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8pxb\" (UniqueName: \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.418357 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418336    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.418430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418398    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.418575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418557    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.418687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418581    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.418687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418599    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.418687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418631    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.418687 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.418644    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.419270 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.419242    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.420114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.420079    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "kube-api-access-dfr4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:09.420244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.420160    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:09.420244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.420190    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:57:09.420392 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.420248    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy" pod "b6b42b50-0b86-484f-9e70-f56ae576c4d0" (UID: "b6b42b50-0b86-484f-9e70-f56ae576c4d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:09.421162 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.421141    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.426825 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.426801    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8pxb\" (UniqueName: \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\") pod \"maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.519620 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.519569    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.519832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.519658    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfr4s\" (UniqueName: \"kubernetes.io/projected/b6b42b50-0b86-484f-9e70-f56ae576c4d0-kube-api-access-dfr4s\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.519832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.519673    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.519832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.519682    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6b42b50-0b86-484f-9e70-f56ae576c4d0-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:09.523156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.523127    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:09.839728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:09.839696    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"]
Oct 31 03:57:09.840313 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:57:09.840286    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd41b05a_bf1c_4a84_b54e_928ad2650994.slice/crio-1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569 WatchSource:0}: Error finding container 1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569: Status 404 returned error can't find the container with id 1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569
Oct 31 03:57:10.131954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.131853    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq" event={"ID":"b6b42b50-0b86-484f-9e70-f56ae576c4d0","Type":"ContainerDied","Data":"78b8e000cd0dd150c4039ffa4c22169b220b666198cc0db20b34545cbb8d9dcc"}
Oct 31 03:57:10.131954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.131927    5098 scope.go:122] "RemoveContainer" containerID="8e18f430341a50f6c639e5c3338507e39cea999cbd2481918611114d5fbfa740"
Oct 31 03:57:10.132425 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.132051    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"
Oct 31 03:57:10.138117 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.138084    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p" event={"ID":"bd41b05a-bf1c-4a84-b54e-928ad2650994","Type":"ContainerStarted","Data":"1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569"}
Oct 31 03:57:10.158537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.158494    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:57:10.162772 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.162740    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-6c6050-openshift-default-7f8bb98f59-kjvzq"]
Oct 31 03:57:10.986777 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:10.986730    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:57:11.153259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:11.153225    5098 generic.go:356] "Generic (PLEG): container finished" podID="bd41b05a-bf1c-4a84-b54e-928ad2650994" containerID="54c15c00c950e306bec9e40da313baed8dfe1321b84d0c3fe5516069a44dcc94" exitCode=0
Oct 31 03:57:11.153816 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:11.153292    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p" event={"ID":"bd41b05a-bf1c-4a84-b54e-928ad2650994","Type":"ContainerDied","Data":"54c15c00c950e306bec9e40da313baed8dfe1321b84d0c3fe5516069a44dcc94"}
Oct 31 03:57:11.618351 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:11.618303    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" path="/var/lib/kubelet/pods/b6b42b50-0b86-484f-9e70-f56ae576c4d0/volumes"
Oct 31 03:57:11.991667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:11.991438    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main" containerID="cri-o://ec755e445033e26b390973b28862803752959e0d45a4069f5694331978bd55ad" gracePeriod=45
Oct 31 03:57:12.166207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.166172    5098 generic.go:356] "Generic (PLEG): container finished" podID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerID="ec755e445033e26b390973b28862803752959e0d45a4069f5694331978bd55ad" exitCode=0
Oct 31 03:57:12.166702 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.166254    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" event={"ID":"bb483b06-2e94-4645-8bdd-3a666a6c4b72","Type":"ContainerDied","Data":"ec755e445033e26b390973b28862803752959e0d45a4069f5694331978bd55ad"}
Oct 31 03:57:12.300295 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.300263    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:57:12.399181 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.399155    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:12.456983 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.456940    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457120    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\" (UniqueName: \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457201 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457162    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457200    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457336 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457213    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:12.457440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457333    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\" (UniqueName: \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457365    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\") pod \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\" (UID: \"bb483b06-2e94-4645-8bdd-3a666a6c4b72\") "
Oct 31 03:57:12.457440 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457424    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:12.457951 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.457924    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.458143 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.458124    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.458304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.458277    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:12.460356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.460304    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:57:12.460356 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.460299    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:12.460498 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.460456    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9" pod "bb483b06-2e94-4645-8bdd-3a666a6c4b72" (UID: "bb483b06-2e94-4645-8bdd-3a666a6c4b72"). InnerVolumeSpecName "kube-api-access-q5wj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:12.559210 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559167    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\") pod \"bd41b05a-bf1c-4a84-b54e-928ad2650994\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") "
Oct 31 03:57:12.559441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559268    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\" (UniqueName: \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\") pod \"bd41b05a-bf1c-4a84-b54e-928ad2650994\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") "
Oct 31 03:57:12.559441 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559326    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\") pod \"bd41b05a-bf1c-4a84-b54e-928ad2650994\" (UID: \"bd41b05a-bf1c-4a84-b54e-928ad2650994\") "
Oct 31 03:57:12.559734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559704    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca" pod "bd41b05a-bf1c-4a84-b54e-928ad2650994" (UID: "bd41b05a-bf1c-4a84-b54e-928ad2650994"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:12.559971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559954    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.560087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559972    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.560087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559983    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wj9\" (UniqueName: \"kubernetes.io/projected/bb483b06-2e94-4645-8bdd-3a666a6c4b72-kube-api-access-q5wj9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.560087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.559994    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bb483b06-2e94-4645-8bdd-3a666a6c4b72-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.560087 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.560007    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.561511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.561479    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca" pod "bd41b05a-bf1c-4a84-b54e-928ad2650994" (UID: "bd41b05a-bf1c-4a84-b54e-928ad2650994"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:57:12.561810 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.561578    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb" pod "bd41b05a-bf1c-4a84-b54e-928ad2650994" (UID: "bd41b05a-bf1c-4a84-b54e-928ad2650994"). InnerVolumeSpecName "kube-api-access-p8pxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:12.661483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.661445    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd41b05a-bf1c-4a84-b54e-928ad2650994-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:12.661483 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:12.661479    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8pxb\" (UniqueName: \"kubernetes.io/projected/bd41b05a-bf1c-4a84-b54e-928ad2650994-kube-api-access-p8pxb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:13.177729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.177698    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"
Oct 31 03:57:13.177729 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.177709    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p" event={"ID":"bd41b05a-bf1c-4a84-b54e-928ad2650994","Type":"ContainerDied","Data":"1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569"}
Oct 31 03:57:13.178271 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.177780    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a76ecf6a50ec59d0ee420305bad3b7d6d0123272a0a57793531de8e3e50a569"
Oct 31 03:57:13.179519 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.179496    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"
Oct 31 03:57:13.179747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.179525    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5" event={"ID":"bb483b06-2e94-4645-8bdd-3a666a6c4b72","Type":"ContainerDied","Data":"aab7dad118e0c526a7a4fbdf77f867caa4d9c1c84dc6887b80cdfb50e661a5e8"}
Oct 31 03:57:13.179747 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.179574    5098 scope.go:122] "RemoveContainer" containerID="ec755e445033e26b390973b28862803752959e0d45a4069f5694331978bd55ad"
Oct 31 03:57:13.237730 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.237697    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:57:13.241975 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.241939    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-090ebd/simulator-w3-090ebd-kserve-b559b5748-22hs5"]
Oct 31 03:57:13.614814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:13.614777    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" path="/var/lib/kubelet/pods/bb483b06-2e94-4645-8bdd-3a666a6c4b72/volumes"
Oct 31 03:57:14.192192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.192112    5098 generic.go:356] "Generic (PLEG): container finished" podID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerID="c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017" exitCode=0
Oct 31 03:57:14.192192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.192166    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerDied","Data":"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"}
Oct 31 03:57:14.192844 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.192822    5098 scope.go:122] "RemoveContainer" containerID="c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"
Oct 31 03:57:14.194945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.194913    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:57:14.195080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.194992    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:57:14.195080 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:14.195027    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 03:57:15.205024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:15.204988    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerStarted","Data":"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"}
Oct 31 03:57:15.714150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:15.709005    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:57:15.714150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:15.711504    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:15.714150 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:15.711587    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:16.708894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:16.708854    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:16.709349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:16.708942    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:17.708146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:17.708103    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:17.708352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:17.708162    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:18.708875 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:18.708836    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:18.709430 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:18.708938    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:19.346226 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:19.346170    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:57:19.713109 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:19.713018    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:19.713763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:19.713725    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:20.708929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:20.708891    5098 patch_prober.go:29] interesting pod/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused" start-of-body=
Oct 31 03:57:20.709159 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:20.708949    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.11:15021/healthz/ready\": dial tcp 10.132.1.11:15021: connect: connection refused"
Oct 31 03:57:20.987252 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:20.987157    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:57:21.713588 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:21.713556    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:57:21.714067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:21.714045    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:57:21.714689 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:21.714669    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:57:23.838353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.838292    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"]
Oct 31 03:57:23.841241 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841205    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main"
Oct 31 03:57:23.841675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841650    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main"
Oct 31 03:57:23.841856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841812    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bd41b05a-bf1c-4a84-b54e-928ad2650994" containerName="revoke-keys"
Oct 31 03:57:23.841856 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841828    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd41b05a-bf1c-4a84-b54e-928ad2650994" containerName="revoke-keys"
Oct 31 03:57:23.842236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841889    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy"
Oct 31 03:57:23.842236 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.841899    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy"
Oct 31 03:57:23.843106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.843084    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd41b05a-bf1c-4a84-b54e-928ad2650994" containerName="revoke-keys"
Oct 31 03:57:23.843312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.843126    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6b42b50-0b86-484f-9e70-f56ae576c4d0" containerName="istio-proxy"
Oct 31 03:57:23.843312 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.843150    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb483b06-2e94-4645-8bdd-3a666a6c4b72" containerName="main"
Oct 31 03:57:23.849954 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.849927    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:23.851657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.851629    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"]
Oct 31 03:57:23.858584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.856853    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:57:23.858584 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.857827    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:57:23.941289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.941237    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:23.941289 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.941295    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxt6b\" (UniqueName: \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:23.941544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:23.941425    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.050086 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.050016    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.056796 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.056742    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxt6b\" (UniqueName: \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.057580 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.057538    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.065700 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.051709    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.066481 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.066396    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.119134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.119053    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxt6b\" (UniqueName: \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\") pod \"maas-api-revoke-sub-6d0da114-x24g5\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.173145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.173108    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:24.250732 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.250695    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"]
Oct 31 03:57:24.257959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.257920    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-090ebd-mxmc7"]
Oct 31 03:57:24.347878 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.347822    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing" containerID="cri-o://e3b0eeeacdf8fd1beb635276f72434db3b90a77204335b39db42e38d73343843" gracePeriod=25
Oct 31 03:57:24.472145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:24.472110    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"]
Oct 31 03:57:24.472649 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:57:24.472618    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef165c6c_c420_444c_a5dc_6c8b304227bb.slice/crio-d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591 WatchSource:0}: Error finding container d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591: Status 404 returned error can't find the container with id d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591
Oct 31 03:57:25.291085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:25.291038    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5" event={"ID":"ef165c6c-c420-444c-a5dc-6c8b304227bb","Type":"ContainerStarted","Data":"f659002372c2394771867cb6bc342dc94aa1e43d417fa7059457dddc7272dd51"}
Oct 31 03:57:25.291569 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:25.291101    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5" event={"ID":"ef165c6c-c420-444c-a5dc-6c8b304227bb","Type":"ContainerStarted","Data":"d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591"}
Oct 31 03:57:25.309792 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:25.309734    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5" podStartSLOduration=2.309717408 podStartE2EDuration="2.309717408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:57:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:57:25.306600068 +0000 UTC m=+2519.831819657" watchObservedRunningTime="2026-10-31 03:57:25.309717408 +0000 UTC m=+2519.834937054"
Oct 31 03:57:25.612038 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:25.611955    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97769d0-2654-4e5a-9111-b127962ae022" path="/var/lib/kubelet/pods/e97769d0-2654-4e5a-9111-b127962ae022/volumes"
Oct 31 03:57:26.301406 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:26.301366    5098 generic.go:356] "Generic (PLEG): container finished" podID="ef165c6c-c420-444c-a5dc-6c8b304227bb" containerID="f659002372c2394771867cb6bc342dc94aa1e43d417fa7059457dddc7272dd51" exitCode=0
Oct 31 03:57:26.301855 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:26.301447    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5" event={"ID":"ef165c6c-c420-444c-a5dc-6c8b304227bb","Type":"ContainerDied","Data":"f659002372c2394771867cb6bc342dc94aa1e43d417fa7059457dddc7272dd51"}
Oct 31 03:57:27.508076 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.508047    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:27.621953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.621873    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\" (UniqueName: \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\") pod \"ef165c6c-c420-444c-a5dc-6c8b304227bb\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") "
Oct 31 03:57:27.622244 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.622218    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\") pod \"ef165c6c-c420-444c-a5dc-6c8b304227bb\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") "
Oct 31 03:57:27.622540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.622521    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\") pod \"ef165c6c-c420-444c-a5dc-6c8b304227bb\" (UID: \"ef165c6c-c420-444c-a5dc-6c8b304227bb\") "
Oct 31 03:57:27.623389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.622637    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca" pod "ef165c6c-c420-444c-a5dc-6c8b304227bb" (UID: "ef165c6c-c420-444c-a5dc-6c8b304227bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:27.623389 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.623358    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:27.629067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.629025    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca" pod "ef165c6c-c420-444c-a5dc-6c8b304227bb" (UID: "ef165c6c-c420-444c-a5dc-6c8b304227bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:57:27.629179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.629086    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b" pod "ef165c6c-c420-444c-a5dc-6c8b304227bb" (UID: "ef165c6c-c420-444c-a5dc-6c8b304227bb"). InnerVolumeSpecName "kube-api-access-qxt6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:27.672549 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.672520    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:57:27.724940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.724900    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxt6b\" (UniqueName: \"kubernetes.io/projected/ef165c6c-c420-444c-a5dc-6c8b304227bb-kube-api-access-qxt6b\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:27.724940 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.724937    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef165c6c-c420-444c-a5dc-6c8b304227bb-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:27.825980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.825936    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\") pod \"25971da9-a3af-42ec-b57a-c36000f39d0e\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") "
Oct 31 03:57:27.826205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.826090    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\" (UniqueName: \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\") pod \"25971da9-a3af-42ec-b57a-c36000f39d0e\" (UID: \"25971da9-a3af-42ec-b57a-c36000f39d0e\") "
Oct 31 03:57:27.826341 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.826316    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume" pod "25971da9-a3af-42ec-b57a-c36000f39d0e" (UID: "25971da9-a3af-42ec-b57a-c36000f39d0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:27.826572 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.826554    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25971da9-a3af-42ec-b57a-c36000f39d0e-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:27.828362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.828339    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6" pod "25971da9-a3af-42ec-b57a-c36000f39d0e" (UID: "25971da9-a3af-42ec-b57a-c36000f39d0e"). InnerVolumeSpecName "kube-api-access-pfpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:27.927734 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:27.927649    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfpc6\" (UniqueName: \"kubernetes.io/projected/25971da9-a3af-42ec-b57a-c36000f39d0e-kube-api-access-pfpc6\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:28.342910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.342868    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"
Oct 31 03:57:28.342910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.342890    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5" event={"ID":"ef165c6c-c420-444c-a5dc-6c8b304227bb","Type":"ContainerDied","Data":"d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591"}
Oct 31 03:57:28.343247 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.342953    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1f5374027cfabf45b4bc3d7d07213e072bdb7346e7d63b0b866688161662591"
Oct 31 03:57:28.344652 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.344587    5098 generic.go:356] "Generic (PLEG): container finished" podID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerID="df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4" exitCode=0
Oct 31 03:57:28.344841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.344688    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"
Oct 31 03:57:28.344841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.344718    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" event={"ID":"25971da9-a3af-42ec-b57a-c36000f39d0e","Type":"ContainerDied","Data":"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"}
Oct 31 03:57:28.344841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.344760    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r" event={"ID":"25971da9-a3af-42ec-b57a-c36000f39d0e","Type":"ContainerDied","Data":"f62d2f4ff572b3f4c470370260632c84edfa8861d29e4b8a9710d38061875bec"}
Oct 31 03:57:28.344841 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.344783    5098 scope.go:122] "RemoveContainer" containerID="df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"
Oct 31 03:57:28.363346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.363319    5098 scope.go:122] "RemoveContainer" containerID="df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"
Oct 31 03:57:28.363741 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:28.363718    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4\": container with ID starting with df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4 not found: ID does not exist" containerID="df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"
Oct 31 03:57:28.363811 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.363754    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4"} err="failed to get container status \"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4\": rpc error: code = NotFound desc = could not find container \"df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4\": container with ID starting with df7435d159447d5b326cbaf85ef018cce79044375579fb7fd534c8ba7a35d0a4 not found: ID does not exist"
Oct 31 03:57:28.393963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.393929    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:57:28.403461 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:28.403420    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-090ebd-6cb865db47-jr75r"]
Oct 31 03:57:29.612478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:29.612443    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" path="/var/lib/kubelet/pods/25971da9-a3af-42ec-b57a-c36000f39d0e/volumes"
Oct 31 03:57:30.986737 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:30.986691    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:57:31.076250 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:31.073441    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:57:31.384642 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:31.384584    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:57:31.384870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:31.384841    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy" containerID="cri-o://d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1" gracePeriod=30
Oct 31 03:57:36.687318 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.687289    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:57:36.811807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.811765    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.811854    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.811906    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812000 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.811957    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812084    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812093    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:36.812192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812129    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812162    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812215    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812306    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\") pod \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\" (UID: \"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf\") "
Oct 31 03:57:36.812411 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812216    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:36.812570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812355    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:36.812570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812418    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:36.812570 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812472    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:36.812910 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812870    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.813054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812914    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.813054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812929    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.813054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812942    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.813054 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.812954    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.814660 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.814631    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "kube-api-access-5hmqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:36.814773 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.814754    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:36.815040 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.815015    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:57:36.815130 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.815011    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy" pod "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" (UID: "a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:36.914504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.914411    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.914504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.914448    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.914504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.914458    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmqt\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-kube-api-access-5hmqt\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:36.914504 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:36.914469    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:37.431936 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.431895    5098 generic.go:356] "Generic (PLEG): container finished" podID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerID="d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1" exitCode=0
Oct 31 03:57:37.432165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.431943    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"
Oct 31 03:57:37.432165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.431981    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" event={"ID":"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf","Type":"ContainerDied","Data":"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"}
Oct 31 03:57:37.432165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.432075    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f" event={"ID":"a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf","Type":"ContainerDied","Data":"06f4b96df038ee3420b716632d30f83b394ce047da3b82e26b2da7b4cf6ff6e9"}
Oct 31 03:57:37.432165 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.432106    5098 scope.go:122] "RemoveContainer" containerID="d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"
Oct 31 03:57:37.445193 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.445174    5098 scope.go:122] "RemoveContainer" containerID="d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"
Oct 31 03:57:37.445530 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:37.445512    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1\": container with ID starting with d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1 not found: ID does not exist" containerID="d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"
Oct 31 03:57:37.445640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.445537    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1"} err="failed to get container status \"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1\": rpc error: code = NotFound desc = could not find container \"d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1\": container with ID starting with d276ecbd6281cd835672094e23badee6ca5664a1cf9535f35ddb578107e03fb1 not found: ID does not exist"
Oct 31 03:57:37.469637 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.469583    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:57:37.473174 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.473145    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-090ebd-openshift-default-7db9664fdc-7jd8f"]
Oct 31 03:57:37.612723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:37.612687    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" path="/var/lib/kubelet/pods/a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf/volumes"
Oct 31 03:57:40.347953 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:40.347456    5098 scope.go:122] "RemoveContainer" containerID="244c3b15958d57c3aeb75e52998d2669a375bec4d20992e6a5ae512f8a7071a8"
Oct 31 03:57:40.986704 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:40.986667    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:57:41.904634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:41.900650    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"]
Oct 31 03:57:41.905093 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:41.905033    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f20dc4-tf46p"]
Oct 31 03:57:43.612993 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:43.612958    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd41b05a-bf1c-4a84-b54e-928ad2650994" path="/var/lib/kubelet/pods/bd41b05a-bf1c-4a84-b54e-928ad2650994/volumes"
Oct 31 03:57:44.494207 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.494169    5098 generic.go:356] "Generic (PLEG): container finished" podID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerID="e3b0eeeacdf8fd1beb635276f72434db3b90a77204335b39db42e38d73343843" exitCode=0
Oct 31 03:57:44.494417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.494244    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" event={"ID":"48accdc9-4ac0-43fe-b087-863bcbb69aa2","Type":"ContainerDied","Data":"e3b0eeeacdf8fd1beb635276f72434db3b90a77204335b39db42e38d73343843"}
Oct 31 03:57:44.655209 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.655181    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:57:44.726276 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.726223    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\") pod \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") "
Oct 31 03:57:44.726809 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.726778    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume" pod "48accdc9-4ac0-43fe-b087-863bcbb69aa2" (UID: "48accdc9-4ac0-43fe-b087-863bcbb69aa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:44.827453 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.827411    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\" (UniqueName: \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\") pod \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\" (UID: \"48accdc9-4ac0-43fe-b087-863bcbb69aa2\") "
Oct 31 03:57:44.827775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.827760    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48accdc9-4ac0-43fe-b087-863bcbb69aa2-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:44.829649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.829589    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms" pod "48accdc9-4ac0-43fe-b087-863bcbb69aa2" (UID: "48accdc9-4ac0-43fe-b087-863bcbb69aa2"). InnerVolumeSpecName "kube-api-access-mlxms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:44.929192 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:44.929087    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlxms\" (UniqueName: \"kubernetes.io/projected/48accdc9-4ac0-43fe-b087-863bcbb69aa2-kube-api-access-mlxms\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:45.507671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.507636    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"
Oct 31 03:57:45.507671 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.507661    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4" event={"ID":"48accdc9-4ac0-43fe-b087-863bcbb69aa2","Type":"ContainerDied","Data":"017251d5e822aeb2be89931f58d912a215d7715a437553b3dc2cbb9c00cd2e9d"}
Oct 31 03:57:45.507929 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.507747    5098 scope.go:122] "RemoveContainer" containerID="e3b0eeeacdf8fd1beb635276f72434db3b90a77204335b39db42e38d73343843"
Oct 31 03:57:45.573031 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.572989    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:57:45.577095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.577063    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f20dc4-b8f9c9cb7-h84v4"]
Oct 31 03:57:45.620900 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:45.620857    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" path="/var/lib/kubelet/pods/48accdc9-4ac0-43fe-b087-863bcbb69aa2/volumes"
Oct 31 03:57:49.626976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:49.626940    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:57:49.627445 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:49.627146    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy" containerID="cri-o://57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426" gracePeriod=30
Oct 31 03:57:53.851585 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:53.851539    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:57:54.966478 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:54.966452    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:57:55.005675 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005630    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.005885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005697    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.005885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005750    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.005885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005803    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.005885 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005853    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.006147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005917    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.006147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005923    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:55.006147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.005953    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.006147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006039    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:55.006147 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006047    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.006400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006158    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\") pod \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\" (UID: \"b25c3543-bb26-44db-8eb0-7a0811d5a63e\") "
Oct 31 03:57:55.006400 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006300    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006698    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006806    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006825    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006837    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006846    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.007052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.006923    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:57:55.008870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.008844    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:55.008870 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.008853    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:57:55.009063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.009009    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:57:55.009063 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.009044    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk" pod "b25c3543-bb26-44db-8eb0-7a0811d5a63e" (UID: "b25c3543-bb26-44db-8eb0-7a0811d5a63e"). InnerVolumeSpecName "kube-api-access-chvkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:57:55.108959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.108915    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.108959 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.108961    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvkk\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-kube-api-access-chvkk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.109213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.108977    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.109213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.109005    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.109213 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.109018    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b25c3543-bb26-44db-8eb0-7a0811d5a63e-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:57:55.596195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.596158    5098 generic.go:356] "Generic (PLEG): container finished" podID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerID="57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426" exitCode=0
Oct 31 03:57:55.596393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.596205    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"
Oct 31 03:57:55.596393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.596230    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerDied","Data":"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"}
Oct 31 03:57:55.596393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.596287    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml" event={"ID":"b25c3543-bb26-44db-8eb0-7a0811d5a63e","Type":"ContainerDied","Data":"c4b8f0dbbf669ce2ccd07f1978fd2f513a9a24f49477f8df7680f74551ab5d74"}
Oct 31 03:57:55.596393 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.596311    5098 scope.go:122] "RemoveContainer" containerID="57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"
Oct 31 03:57:55.608917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.608897    5098 scope.go:122] "RemoveContainer" containerID="347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"
Oct 31 03:57:55.624122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.624097    5098 scope.go:122] "RemoveContainer" containerID="57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"
Oct 31 03:57:55.625908 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.625877    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:57:55.627747 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:55.627708    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426\": container with ID starting with 57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426 not found: ID does not exist" containerID="57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"
Oct 31 03:57:55.627862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.627759    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426"} err="failed to get container status \"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426\": rpc error: code = NotFound desc = could not find container \"57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426\": container with ID starting with 57da19eff15ebfcd6d197034bb7c2e313be37121fe0f2c5524cb782cb6747426 not found: ID does not exist"
Oct 31 03:57:55.627862 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.627785    5098 scope.go:122] "RemoveContainer" containerID="347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"
Oct 31 03:57:55.628134 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:57:55.628110    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444\": container with ID starting with 347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444 not found: ID does not exist" containerID="347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"
Oct 31 03:57:55.628183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.628146    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444"} err="failed to get container status \"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444\": rpc error: code = NotFound desc = could not find container \"347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444\": container with ID starting with 347ae2e4a5ba204d1b59e67fca09726c776fa4d01da00965a0fb08aa9c040444 not found: ID does not exist"
Oct 31 03:57:55.631640 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:55.631616    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f20dc4-openshift-default-588d68b56c-xb7ml"]
Oct 31 03:57:57.612783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:57:57.612739    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" path="/var/lib/kubelet/pods/b25c3543-bb26-44db-8eb0-7a0811d5a63e/volumes"
Oct 31 03:58:01.057934 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:58:01.057894    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:58:24.527074 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:58:24.527035    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:58:31.061810 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:58:31.061766    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:58:36.546140 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:58:36.546100    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:58:52.049853 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:58:52.049809    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:58:55.470088 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:58:55.470046    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:58:56.854085 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:58:56.854048    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:59:01.058165 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:01.058125    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:59:07.074567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.074495    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" podUID="67f732fb-d44d-467d-942e-11431c60f267" containerName="main" containerID="cri-o://130c68a23c8a52615f53d944b35ed7672802ee1a48632d2757c081be90e839f3" gracePeriod=45
Oct 31 03:59:07.131485 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.131436    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" podUID="67f732fb-d44d-467d-942e-11431c60f267" containerName="main" probeResult="failure" output="Get \"https://10.132.1.23:8000/ready\": dial tcp 10.132.1.23:8000: connect: connection refused"
Oct 31 03:59:07.220432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.220392    5098 generic.go:356] "Generic (PLEG): container finished" podID="67f732fb-d44d-467d-942e-11431c60f267" containerID="130c68a23c8a52615f53d944b35ed7672802ee1a48632d2757c081be90e839f3" exitCode=0
Oct 31 03:59:07.220631 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.220461    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" event={"ID":"67f732fb-d44d-467d-942e-11431c60f267","Type":"ContainerDied","Data":"130c68a23c8a52615f53d944b35ed7672802ee1a48632d2757c081be90e839f3"}
Oct 31 03:59:07.373156 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.373124    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:59:07.522641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.522593    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.522827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.522693    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\" (UniqueName: \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.522827 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.522752    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.522931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.522882    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.522931 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.522894    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:07.523025 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523012    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\" (UniqueName: \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.523104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523057    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\") pod \"67f732fb-d44d-467d-942e-11431c60f267\" (UID: \"67f732fb-d44d-467d-942e-11431c60f267\") "
Oct 31 03:59:07.523221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523102    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:07.523363 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523325    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:07.523638 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523620    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:07.523728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523642    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:07.523728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.523658    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:07.525034 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.525013    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:07.525131 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.525110    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "kube-api-access-25l5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:07.525225 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.525209    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs" pod "67f732fb-d44d-467d-942e-11431c60f267" (UID: "67f732fb-d44d-467d-942e-11431c60f267"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:07.625122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.625093    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25l5q\" (UniqueName: \"kubernetes.io/projected/67f732fb-d44d-467d-942e-11431c60f267-kube-api-access-25l5q\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:07.625122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.625122    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67f732fb-d44d-467d-942e-11431c60f267-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:07.625298 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:07.625133    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67f732fb-d44d-467d-942e-11431c60f267-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:08.023379 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.023286    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"]
Oct 31 03:59:08.026806 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.026777    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f603a02-h6s8d"]
Oct 31 03:59:08.229575 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.229544    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"
Oct 31 03:59:08.230128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.229581    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6" event={"ID":"67f732fb-d44d-467d-942e-11431c60f267","Type":"ContainerDied","Data":"1727edeae732654e07366d32cfd0415191def3a8518a5f06c205b4484ae0d5d5"}
Oct 31 03:59:08.230128 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.229655    5098 scope.go:122] "RemoveContainer" containerID="130c68a23c8a52615f53d944b35ed7672802ee1a48632d2757c081be90e839f3"
Oct 31 03:59:08.250405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.250374    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:59:08.253723 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:08.253698    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w6-20585c/auth-test-model-20585c-kserve-7f8bf85bb5-pxbp6"]
Oct 31 03:59:09.024421 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.024385    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"]
Oct 31 03:59:09.029056 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.029019    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"]
Oct 31 03:59:09.045019 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.044968    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4226c74-6lvvj"]
Oct 31 03:59:09.048428 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.048403    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0268511-qcftz"]
Oct 31 03:59:09.613183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.613140    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67f732fb-d44d-467d-942e-11431c60f267" path="/var/lib/kubelet/pods/67f732fb-d44d-467d-942e-11431c60f267/volumes"
Oct 31 03:59:09.613657 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.613642    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75979d73-8cc2-486e-9fec-de3e54c076df" path="/var/lib/kubelet/pods/75979d73-8cc2-486e-9fec-de3e54c076df/volumes"
Oct 31 03:59:09.614094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.614079    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04c4f7c-074a-4f49-a4f3-59d7a8063f65" path="/var/lib/kubelet/pods/c04c4f7c-074a-4f49-a4f3-59d7a8063f65/volumes"
Oct 31 03:59:09.614742 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:09.614720    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe547e8d-37d6-41cb-b99f-10b0c9218292" path="/var/lib/kubelet/pods/fe547e8d-37d6-41cb-b99f-10b0c9218292/volumes"
Oct 31 03:59:11.876196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:11.876138    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" podUID="ae2082e8-d563-408f-a911-12646033a523" containerName="main" containerID="cri-o://2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3" gracePeriod=45
Oct 31 03:59:12.184788 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.184758    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:59:12.263094 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.263059    5098 generic.go:356] "Generic (PLEG): container finished" podID="ae2082e8-d563-408f-a911-12646033a523" containerID="2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3" exitCode=0
Oct 31 03:59:12.263325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.263126    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"
Oct 31 03:59:12.263325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.263149    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" event={"ID":"ae2082e8-d563-408f-a911-12646033a523","Type":"ContainerDied","Data":"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"}
Oct 31 03:59:12.263325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.263202    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8" event={"ID":"ae2082e8-d563-408f-a911-12646033a523","Type":"ContainerDied","Data":"49eb1996b410413968d54f6da08964296068631c058ca23f181088318d9ff4c9"}
Oct 31 03:59:12.263325 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.263226    5098 scope.go:122] "RemoveContainer" containerID="2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"
Oct 31 03:59:12.274294 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.274273    5098 scope.go:122] "RemoveContainer" containerID="2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"
Oct 31 03:59:12.274666 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:12.274641    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3\": container with ID starting with 2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3 not found: ID does not exist" containerID="2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"
Oct 31 03:59:12.274731 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.274676    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3"} err="failed to get container status \"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3\": rpc error: code = NotFound desc = could not find container \"2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3\": container with ID starting with 2709f2284466d9218571290734d6772bc205ee320d563d7175d7b20ebd20dcc3 not found: ID does not exist"
Oct 31 03:59:12.285163 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285124    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285188    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285240    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285343 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285269    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\" (UniqueName: \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285340    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285445    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.285515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285456    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\") pod \"ae2082e8-d563-408f-a911-12646033a523\" (UID: \"ae2082e8-d563-408f-a911-12646033a523\") "
Oct 31 03:59:12.285515 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285497    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.285775 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.285719    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.286067 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.286040    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.286188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.286070    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.286188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.286085    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.287672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.287647    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:12.287771 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.287707    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "kube-api-access-98hf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:12.287832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.287808    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs" pod "ae2082e8-d563-408f-a911-12646033a523" (UID: "ae2082e8-d563-408f-a911-12646033a523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:12.369634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.369575    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:59:12.387914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.387879    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae2082e8-d563-408f-a911-12646033a523-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.387914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.387912    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98hf8\" (UniqueName: \"kubernetes.io/projected/ae2082e8-d563-408f-a911-12646033a523-kube-api-access-98hf8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.387914 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.387924    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae2082e8-d563-408f-a911-12646033a523-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:12.587807 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.587759    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:59:12.592587 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:12.592553    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-39e8cf/auth-test-model-39e8cf-kserve-65d599f47c-xzfk8"]
Oct 31 03:59:13.612021 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:13.611986    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2082e8-d563-408f-a911-12646033a523" path="/var/lib/kubelet/pods/ae2082e8-d563-408f-a911-12646033a523/volumes"
Oct 31 03:59:15.935118 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:15.935076    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:59:15.935667 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:15.935294    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy" containerID="cri-o://4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3" gracePeriod=30
Oct 31 03:59:15.939203 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:15.939170    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:59:15.939722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:15.939676    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy" containerID="cri-o://c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042" gracePeriod=30
Oct 31 03:59:21.267177 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.267144    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:59:21.285346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.285317    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:59:21.343726 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.343689    5098 generic.go:356] "Generic (PLEG): container finished" podID="164f671b-b44e-4893-8b64-51baf403ba2e" containerID="c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042" exitCode=0
Oct 31 03:59:21.343915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.343741    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"
Oct 31 03:59:21.343915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.343770    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" event={"ID":"164f671b-b44e-4893-8b64-51baf403ba2e","Type":"ContainerDied","Data":"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"}
Oct 31 03:59:21.343915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.343837    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz" event={"ID":"164f671b-b44e-4893-8b64-51baf403ba2e","Type":"ContainerDied","Data":"f32667a0a7e2508dac98df72881e758e45f416f182adfc0fd04edf4533c95f33"}
Oct 31 03:59:21.343915 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.343863    5098 scope.go:122] "RemoveContainer" containerID="c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"
Oct 31 03:59:21.345669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.345642    5098 generic.go:356] "Generic (PLEG): container finished" podID="11027739-4566-428c-b85b-c84ae5513314" containerID="4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3" exitCode=0
Oct 31 03:59:21.345794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.345687    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"
Oct 31 03:59:21.345794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.345697    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" event={"ID":"11027739-4566-428c-b85b-c84ae5513314","Type":"ContainerDied","Data":"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"}
Oct 31 03:59:21.345794 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.345754    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk" event={"ID":"11027739-4566-428c-b85b-c84ae5513314","Type":"ContainerDied","Data":"6711933cb91e12d6786412128ab3e7850eb8e0e514116d7bd3f8501d1483617f"}
Oct 31 03:59:21.351701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.351664    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.351814 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.351755    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.351864 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.351829    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.351917 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.351886    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.351970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.351947    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352024 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352008    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352019    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.352092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352067    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352106    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352141    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352184    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352211    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352241    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352272    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352298    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352306    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352354    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:21.352382 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352360    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352436    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352473    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\") pod \"11027739-4566-428c-b85b-c84ae5513314\" (UID: \"11027739-4566-428c-b85b-c84ae5513314\") "
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352506    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352509    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\") pod \"164f671b-b44e-4893-8b64-51baf403ba2e\" (UID: \"164f671b-b44e-4893-8b64-51baf403ba2e\") "
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352598    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.352832 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.352718    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.353229 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353017    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:21.353296 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353243    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.353370 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353344    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11027739-4566-428c-b85b-c84ae5513314-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353376    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353391    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353405    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/164f671b-b44e-4893-8b64-51baf403ba2e-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353419    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353433    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.353756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353442    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.353756 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.353447    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.354641 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.354538    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.354805 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.354784    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.356284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356187    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:21.356284 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356247    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:21.356415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356283    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "kube-api-access-swdk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.356415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356288    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.356415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356308    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp" pod "11027739-4566-428c-b85b-c84ae5513314" (UID: "11027739-4566-428c-b85b-c84ae5513314"). InnerVolumeSpecName "kube-api-access-b48gp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.356415 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356406    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:21.356567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.356553    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token" pod "164f671b-b44e-4893-8b64-51baf403ba2e" (UID: "164f671b-b44e-4893-8b64-51baf403ba2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:21.365536 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.365510    5098 scope.go:122] "RemoveContainer" containerID="c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"
Oct 31 03:59:21.365925 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:21.365902    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042\": container with ID starting with c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042 not found: ID does not exist" containerID="c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"
Oct 31 03:59:21.365979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.365936    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042"} err="failed to get container status \"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042\": rpc error: code = NotFound desc = could not find container \"c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042\": container with ID starting with c85db43cbffbded677dd8a98e849348163241d27390136c8a493edf66c686042 not found: ID does not exist"
Oct 31 03:59:21.365979 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.365958    5098 scope.go:122] "RemoveContainer" containerID="4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"
Oct 31 03:59:21.376751 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.376729    5098 scope.go:122] "RemoveContainer" containerID="4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"
Oct 31 03:59:21.377103 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:21.377082    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3\": container with ID starting with 4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3 not found: ID does not exist" containerID="4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"
Oct 31 03:59:21.377161 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.377112    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3"} err="failed to get container status \"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3\": rpc error: code = NotFound desc = could not find container \"4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3\": container with ID starting with 4bce28bff646c1a981e2fc42c31b52dd914af3b6b1ff3fb8244bae6f87d96ac3 not found: ID does not exist"
Oct 31 03:59:21.455259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455213    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/164f671b-b44e-4893-8b64-51baf403ba2e-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455250    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455259 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455261    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48gp\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-kube-api-access-b48gp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455271    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11027739-4566-428c-b85b-c84ae5513314-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455280    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455288    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11027739-4566-428c-b85b-c84ae5513314-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455297    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swdk2\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-kube-api-access-swdk2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455306    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/164f671b-b44e-4893-8b64-51baf403ba2e-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455314    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11027739-4566-428c-b85b-c84ae5513314-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455322    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.455540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.455330    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/164f671b-b44e-4893-8b64-51baf403ba2e-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:21.664718 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.664680    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:59:21.668998 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.668970    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-sw4tk"]
Oct 31 03:59:21.681795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.681761    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:59:21.686795 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.686767    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-20585c-openshift-default-55664fb769-p7fmz"]
Oct 31 03:59:21.993037 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:21.992999    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:59:23.613529 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:23.613490    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11027739-4566-428c-b85b-c84ae5513314" path="/var/lib/kubelet/pods/11027739-4566-428c-b85b-c84ae5513314/volumes"
Oct 31 03:59:23.614202 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:23.614182    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" path="/var/lib/kubelet/pods/164f671b-b44e-4893-8b64-51baf403ba2e/volumes"
Oct 31 03:59:26.630511 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:26.630474    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:59:26.631768 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:26.631735    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:59:30.021868 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.021823    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"]
Oct 31 03:59:30.024926 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.024896    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b10398d-qjsvz"]
Oct 31 03:59:30.314753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.309053    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:59:30.314753 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.309328    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" podUID="91926258-2c8b-4918-980e-d066b6b68843" containerName="maas-api" containerID="cri-o://fc7aadb742efbca38406d79eac2730e506ecb5a68487885fa68dc9eefd2fb53c" gracePeriod=30
Oct 31 03:59:30.453906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.453869    5098 generic.go:356] "Generic (PLEG): container finished" podID="91926258-2c8b-4918-980e-d066b6b68843" containerID="fc7aadb742efbca38406d79eac2730e506ecb5a68487885fa68dc9eefd2fb53c" exitCode=0
Oct 31 03:59:30.454096 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.453941    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" event={"ID":"91926258-2c8b-4918-980e-d066b6b68843","Type":"ContainerDied","Data":"fc7aadb742efbca38406d79eac2730e506ecb5a68487885fa68dc9eefd2fb53c"}
Oct 31 03:59:30.621647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.609966    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:59:30.700082 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700043    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\") pod \"91926258-2c8b-4918-980e-d066b6b68843\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") "
Oct 31 03:59:30.700308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700108    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\") pod \"91926258-2c8b-4918-980e-d066b6b68843\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") "
Oct 31 03:59:30.700308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700280    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\" (UniqueName: \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\") pod \"91926258-2c8b-4918-980e-d066b6b68843\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") "
Oct 31 03:59:30.700308 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700303    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\") pod \"91926258-2c8b-4918-980e-d066b6b68843\" (UID: \"91926258-2c8b-4918-980e-d066b6b68843\") "
Oct 31 03:59:30.700542 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700514    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca" pod "91926258-2c8b-4918-980e-d066b6b68843" (UID: "91926258-2c8b-4918-980e-d066b6b68843"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:30.700745 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.700728    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91926258-2c8b-4918-980e-d066b6b68843-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.702555 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.702526    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7" pod "91926258-2c8b-4918-980e-d066b6b68843" (UID: "91926258-2c8b-4918-980e-d066b6b68843"). InnerVolumeSpecName "kube-api-access-26sg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:30.702669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.702562    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls" pod "91926258-2c8b-4918-980e-d066b6b68843" (UID: "91926258-2c8b-4918-980e-d066b6b68843"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:30.702669 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.702584    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs" pod "91926258-2c8b-4918-980e-d066b6b68843" (UID: "91926258-2c8b-4918-980e-d066b6b68843"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:30.802388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.802340    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26sg7\" (UniqueName: \"kubernetes.io/projected/91926258-2c8b-4918-980e-d066b6b68843-kube-api-access-26sg7\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.802388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.802383    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:30.802388 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:30.802394    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91926258-2c8b-4918-980e-d066b6b68843-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:31.060700 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:31.060646    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 03:59:31.463681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.463564    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"
Oct 31 03:59:31.463681 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.463596    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8" event={"ID":"91926258-2c8b-4918-980e-d066b6b68843","Type":"ContainerDied","Data":"2d56b17e0688274ed549eb42b28a8c365909415d69f8b272f72c5ebed711d404"}
Oct 31 03:59:31.463895 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.463705    5098 scope.go:122] "RemoveContainer" containerID="fc7aadb742efbca38406d79eac2730e506ecb5a68487885fa68dc9eefd2fb53c"
Oct 31 03:59:31.488790 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.488732    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:59:31.493577 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.493541    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-39e8cf-59b8458d59-vknl8"]
Oct 31 03:59:31.612184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.612149    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91926258-2c8b-4918-980e-d066b6b68843" path="/var/lib/kubelet/pods/91926258-2c8b-4918-980e-d066b6b68843/volumes"
Oct 31 03:59:31.612728 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.612711    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d107debb-0e4d-4c39-9d8e-5c8e6cfa9401" path="/var/lib/kubelet/pods/d107debb-0e4d-4c39-9d8e-5c8e6cfa9401/volumes"
Oct 31 03:59:31.631248 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.631176    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing" containerID="cri-o://2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512" gracePeriod=25
Oct 31 03:59:31.632722 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.632671    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing" containerID="cri-o://8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9" gracePeriod=25
Oct 31 03:59:31.870079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.870038    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:31.870079 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:31.870061    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:33.136616 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:33.136561    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:59:37.786703 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:37.786556    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:59:37.787221 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:37.786847    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy" containerID="cri-o://52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf" gracePeriod=30
Oct 31 03:59:38.931417 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:38.931374    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:59:38.931970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:38.931688    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy" containerID="cri-o://1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956" gracePeriod=30
Oct 31 03:59:39.489581 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.489533    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"]
Oct 31 03:59:39.490144 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490127    5098 cpu_manager.go:414] "containerMap: removing container" podUID="91926258-2c8b-4918-980e-d066b6b68843" containerName="maas-api"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490146    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91926258-2c8b-4918-980e-d066b6b68843" containerName="maas-api"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490160    5098 cpu_manager.go:414] "containerMap: removing container" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490165    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490181    5098 cpu_manager.go:414] "containerMap: removing container" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490188    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490197    5098 cpu_manager.go:414] "containerMap: removing container" podUID="67f732fb-d44d-467d-942e-11431c60f267" containerName="main"
Oct 31 03:59:39.490205 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490202    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67f732fb-d44d-467d-942e-11431c60f267" containerName="main"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490219    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ef165c6c-c420-444c-a5dc-6c8b304227bb" containerName="revoke-keys"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490225    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef165c6c-c420-444c-a5dc-6c8b304227bb" containerName="revoke-keys"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490235    5098 cpu_manager.go:414] "containerMap: removing container" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490241    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490256    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490261    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490268    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ae2082e8-d563-408f-a911-12646033a523" containerName="main"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490273    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2082e8-d563-408f-a911-12646033a523" containerName="main"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490281    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490286    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490294    5098 cpu_manager.go:414] "containerMap: removing container" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490299    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490306    5098 cpu_manager.go:414] "containerMap: removing container" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490311    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490387    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0cb7e52-bfbf-4fb0-a5d6-ad9a17f89edf" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490397    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="48accdc9-4ac0-43fe-b087-863bcbb69aa2" containerName="payload-pre-processing"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490407    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="67f732fb-d44d-467d-942e-11431c60f267" containerName="main"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490414    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef165c6c-c420-444c-a5dc-6c8b304227bb" containerName="revoke-keys"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490420    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="164f671b-b44e-4893-8b64-51baf403ba2e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490428    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490434    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2082e8-d563-408f-a911-12646033a523" containerName="main"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490438    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25c3543-bb26-44db-8eb0-7a0811d5a63e" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490445    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="11027739-4566-428c-b85b-c84ae5513314" containerName="istio-proxy"
Oct 31 03:59:39.490442 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490452    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="25971da9-a3af-42ec-b57a-c36000f39d0e" containerName="payload-pre-processing"
Oct 31 03:59:39.491701 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.490457    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="91926258-2c8b-4918-980e-d066b6b68843" containerName="maas-api"
Oct 31 03:59:39.495647 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.495597    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.501282 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.501255    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:59:39.501537 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.501341    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:59:39.501863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.501831    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"]
Oct 31 03:59:39.505028 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.505006    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:59:39.505175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.505117    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:59:39.522190 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.522145    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.522368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.522201    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b57j4\" (UniqueName: \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.522368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.522305    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.627299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.623653    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.627299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.623720    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b57j4\" (UniqueName: \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.627299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.623790    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.627299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.625579    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.632346 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.632311    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.635590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.635560    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b57j4\" (UniqueName: \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:39.817158 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:39.817114    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:40.116011 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:40.115943    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"]
Oct 31 03:59:40.117702 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:59:40.117669    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52ac339_072a_4ef1_b39a_bfa4fe27a844.slice/crio-a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3 WatchSource:0}: Error finding container a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3: Status 404 returned error can't find the container with id a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3
Oct 31 03:59:40.558698 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:40.558660    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d" event={"ID":"c52ac339-072a-4ef1-b39a-bfa4fe27a844","Type":"ContainerStarted","Data":"61c3e8be68eeb495f44e56e1eb93e7598b732cb2885fcbcfeec2b02fc757b327"}
Oct 31 03:59:40.558894 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:40.558729    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d" event={"ID":"c52ac339-072a-4ef1-b39a-bfa4fe27a844","Type":"ContainerStarted","Data":"a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3"}
Oct 31 03:59:40.578554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:40.578256    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d" podStartSLOduration=1.5782347730000001 podStartE2EDuration="1.578234773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:40.575870006 +0000 UTC m=+2655.101089598" watchObservedRunningTime="2026-10-31 03:59:40.578234773 +0000 UTC m=+2655.103454363"
Oct 31 03:59:41.569048 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:41.569003    5098 generic.go:356] "Generic (PLEG): container finished" podID="c52ac339-072a-4ef1-b39a-bfa4fe27a844" containerID="61c3e8be68eeb495f44e56e1eb93e7598b732cb2885fcbcfeec2b02fc757b327" exitCode=0
Oct 31 03:59:41.569470 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:41.569076    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d" event={"ID":"c52ac339-072a-4ef1-b39a-bfa4fe27a844","Type":"ContainerDied","Data":"61c3e8be68eeb495f44e56e1eb93e7598b732cb2885fcbcfeec2b02fc757b327"}
Oct 31 03:59:41.870146 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:41.870062    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:41.870304 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:41.870062    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:42.750490 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.750460    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:42.861851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.861748    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\") pod \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") "
Oct 31 03:59:42.861851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.861820    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\" (UniqueName: \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\") pod \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") "
Oct 31 03:59:42.862104 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.861863    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\") pod \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\" (UID: \"c52ac339-072a-4ef1-b39a-bfa4fe27a844\") "
Oct 31 03:59:42.862176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.862148    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca" pod "c52ac339-072a-4ef1-b39a-bfa4fe27a844" (UID: "c52ac339-072a-4ef1-b39a-bfa4fe27a844"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:42.862368 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.862346    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:42.863991 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.863970    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca" pod "c52ac339-072a-4ef1-b39a-bfa4fe27a844" (UID: "c52ac339-072a-4ef1-b39a-bfa4fe27a844"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:42.864124 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.864102    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4" pod "c52ac339-072a-4ef1-b39a-bfa4fe27a844" (UID: "c52ac339-072a-4ef1-b39a-bfa4fe27a844"). InnerVolumeSpecName "kube-api-access-b57j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:42.964169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.964127    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b57j4\" (UniqueName: \"kubernetes.io/projected/c52ac339-072a-4ef1-b39a-bfa4fe27a844-kube-api-access-b57j4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:42.964169 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:42.964173    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c52ac339-072a-4ef1-b39a-bfa4fe27a844-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.056557 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.056530    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:59:43.166256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166165    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166256 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166215    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166265    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166351    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166410    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166441    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166512 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166487    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166580    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166597    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:43.166877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166652    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:43.166877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166680    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\") pod \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\" (UID: \"5e2df149-41c5-41c9-aeaf-2bd567db9cf0\") "
Oct 31 03:59:43.166877 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166776    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:43.167170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.166971    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:43.167230 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.167171    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.167291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.167255    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:43.167291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.167191    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.167405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.167304    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.167405 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.167319    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.168949 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.168921    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:43.169081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.169005    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "kube-api-access-9wvw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:43.169649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.169584    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:43.169649 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.169620    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token" pod "5e2df149-41c5-41c9-aeaf-2bd567db9cf0" (UID: "5e2df149-41c5-41c9-aeaf-2bd567db9cf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:43.268925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.268885    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wvw9\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-kube-api-access-9wvw9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.268925 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.268923    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.269172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.268936    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.269172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.268949    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.269172 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.268963    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e2df149-41c5-41c9-aeaf-2bd567db9cf0-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:43.270891 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.270860    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:59:43.602145 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.602110    5098 generic.go:356] "Generic (PLEG): container finished" podID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerID="52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf" exitCode=0
Oct 31 03:59:43.602362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.602169    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"
Oct 31 03:59:43.602362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.602197    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerDied","Data":"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"}
Oct 31 03:59:43.602362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.602294    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng" event={"ID":"5e2df149-41c5-41c9-aeaf-2bd567db9cf0","Type":"ContainerDied","Data":"e8ca9a0d4213336ee43e993e0396f321177ba5de70717f17fbea6914ad6fe747"}
Oct 31 03:59:43.602362 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.602325    5098 scope.go:122] "RemoveContainer" containerID="52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"
Oct 31 03:59:43.604830 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.604808    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"
Oct 31 03:59:43.617010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.616973    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d" event={"ID":"c52ac339-072a-4ef1-b39a-bfa4fe27a844","Type":"ContainerDied","Data":"a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3"}
Oct 31 03:59:43.617155 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.617034    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6b6ddfd7b631e6848c5da95f2890fd8ac82e53efd08e04e88c8c3da5f3b57a3"
Oct 31 03:59:43.617508 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.617491    5098 scope.go:122] "RemoveContainer" containerID="c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"
Oct 31 03:59:43.633119 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.633094    5098 scope.go:122] "RemoveContainer" containerID="52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"
Oct 31 03:59:43.633497 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:43.633474    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf\": container with ID starting with 52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf not found: ID does not exist" containerID="52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"
Oct 31 03:59:43.633544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.633510    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf"} err="failed to get container status \"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf\": rpc error: code = NotFound desc = could not find container \"52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf\": container with ID starting with 52cd09ab0c1d7daa0a479020e475e0aeade196febb143f3a063c57dce01089cf not found: ID does not exist"
Oct 31 03:59:43.633544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.633530    5098 scope.go:122] "RemoveContainer" containerID="c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"
Oct 31 03:59:43.633895 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:43.633871    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017\": container with ID starting with c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017 not found: ID does not exist" containerID="c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"
Oct 31 03:59:43.633963 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.633901    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017"} err="failed to get container status \"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017\": rpc error: code = NotFound desc = could not find container \"c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017\": container with ID starting with c8afd9ac7dd676ae76fbb4c49a99d69d59bda97522feacc5acd8c391cd092017 not found: ID does not exist"
Oct 31 03:59:43.636114 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.636086    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:59:43.639896 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:43.639872    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-39e8cf-openshift-default-67bfc9844f-lcnng"]
Oct 31 03:59:44.213590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.213518    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:59:44.288332 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.288285    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.288659 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.288601    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.288843 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.288826    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289010 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.288995    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289188 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.289168    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.289339    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289602 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.289583    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289762 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.289746    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.289906 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.289892    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\") pod \"42b27600-cbef-4ca0-9818-265b7b550325\" (UID: \"42b27600-cbef-4ca0-9818-265b7b550325\") "
Oct 31 03:59:44.290971 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.290943    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:44.292211 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.291751    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:44.298818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.298688    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:44.298818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.298782    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:44.298818 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.298778    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 03:59:44.306920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.298867    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:44.306920 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.299279    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:44.307682 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.307598    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "kube-api-access-5gl5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:44.308309 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.308263    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token" pod "42b27600-cbef-4ca0-9818-265b7b550325" (UID: "42b27600-cbef-4ca0-9818-265b7b550325"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:44.391776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391739    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.391776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391770    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.391776 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391780    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391789    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/42b27600-cbef-4ca0-9818-265b7b550325-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391798    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391809    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gl5w\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-kube-api-access-5gl5w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391819    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/42b27600-cbef-4ca0-9818-265b7b550325-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391827    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/42b27600-cbef-4ca0-9818-265b7b550325-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.392050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.391837    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/42b27600-cbef-4ca0-9818-265b7b550325-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:44.614489 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.614446    5098 generic.go:356] "Generic (PLEG): container finished" podID="42b27600-cbef-4ca0-9818-265b7b550325" containerID="1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956" exitCode=0
Oct 31 03:59:44.614721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.614503    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"
Oct 31 03:59:44.614721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.614529    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerDied","Data":"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"}
Oct 31 03:59:44.614721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.614592    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk" event={"ID":"42b27600-cbef-4ca0-9818-265b7b550325","Type":"ContainerDied","Data":"18080a53fe66a519b6c7a23c8a85848d57199b3eee6353ebe71951428db73fb9"}
Oct 31 03:59:44.614721 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.614659    5098 scope.go:122] "RemoveContainer" containerID="1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"
Oct 31 03:59:44.626594 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.626573    5098 scope.go:122] "RemoveContainer" containerID="aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"
Oct 31 03:59:44.637544 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.637517    5098 scope.go:122] "RemoveContainer" containerID="1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"
Oct 31 03:59:44.637943 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:44.637910    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956\": container with ID starting with 1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956 not found: ID does not exist" containerID="1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"
Oct 31 03:59:44.638060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.637953    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956"} err="failed to get container status \"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956\": rpc error: code = NotFound desc = could not find container \"1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956\": container with ID starting with 1817de2e750bb2f2354898d8e32e64eb97c23d60945f1df99d58e78184420956 not found: ID does not exist"
Oct 31 03:59:44.638060 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.637980    5098 scope.go:122] "RemoveContainer" containerID="aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"
Oct 31 03:59:44.638292 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:44.638271    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a\": container with ID starting with aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a not found: ID does not exist" containerID="aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"
Oct 31 03:59:44.638338 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.638298    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a"} err="failed to get container status \"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a\": rpc error: code = NotFound desc = could not find container \"aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a\": container with ID starting with aafced93c08b2e4eb4a69ab7929884218875b815fc54a936bd0888d8c1cc890a not found: ID does not exist"
Oct 31 03:59:44.639711 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.639688    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:59:44.644713 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:44.644687    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-34cb76-openshift-default-5cfcd778ff-9wkkk"]
Oct 31 03:59:45.618179 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:45.618142    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b27600-cbef-4ca0-9818-265b7b550325" path="/var/lib/kubelet/pods/42b27600-cbef-4ca0-9818-265b7b550325/volumes"
Oct 31 03:59:45.618673 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:45.618657    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" path="/var/lib/kubelet/pods/5e2df149-41c5-41c9-aeaf-2bd567db9cf0/volumes"
Oct 31 03:59:48.593909 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:48.593857    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 03:59:48.600743 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:48.600703    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 03:59:49.594945 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:49.594905    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 03:59:50.462068 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462026    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"]
Oct 31 03:59:50.462590 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462576    5098 cpu_manager.go:414] "containerMap: removing container" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462592    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462625    5098 cpu_manager.go:414] "containerMap: removing container" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462634    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462675    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462691 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462683    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462695    5098 cpu_manager.go:414] "containerMap: removing container" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462701    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462713    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c52ac339-072a-4ef1-b39a-bfa4fe27a844" containerName="revoke-keys"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462718    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52ac339-072a-4ef1-b39a-bfa4fe27a844" containerName="revoke-keys"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462835    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462844    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b27600-cbef-4ca0-9818-265b7b550325" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462854    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462860    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e2df149-41c5-41c9-aeaf-2bd567db9cf0" containerName="istio-proxy"
Oct 31 03:59:50.462947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.462868    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52ac339-072a-4ef1-b39a-bfa4fe27a844" containerName="revoke-keys"
Oct 31 03:59:50.468233 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.468203    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.475516 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.475481    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"]
Oct 31 03:59:50.563184 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.563152    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 03:59:50.563352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.563251    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 03:59:50.563352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.563284    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 03:59:50.563352 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.563321    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 03:59:50.564851 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.564830    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.564930 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.564904    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.564976 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.564964    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpcb7\" (UniqueName: \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.666567 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.666520    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.667122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.666681    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.667122 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.666724    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpcb7\" (UniqueName: \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.667353 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.667334    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.669433 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.669404    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.674934 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.674906    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpcb7\" (UniqueName: \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\") pod \"maas-api-revoke-sub-38ccbea0-676lb\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:50.791349 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:50.791310    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:51.022052 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.022016    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"]
Oct 31 03:59:51.024996 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.024962    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d9fae4b-55krz"]
Oct 31 03:59:51.073947 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.073917    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"]
Oct 31 03:59:51.074923 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:59:51.074896    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29b368a6_341b_40aa_b072_aee1a1427533.slice/crio-a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15 WatchSource:0}: Error finding container a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15: Status 404 returned error can't find the container with id a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15
Oct 31 03:59:51.619893 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.618981    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d960d4bd-772c-41aa-b505-b97b38677413" path="/var/lib/kubelet/pods/d960d4bd-772c-41aa-b505-b97b38677413/volumes"
Oct 31 03:59:51.676006 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.675967    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb" event={"ID":"29b368a6-341b-40aa-b072-aee1a1427533","Type":"ContainerStarted","Data":"1560671dca9a86100188de39923e351e96e9a0cc9a28fbd4c58fac392bed9f5b"}
Oct 31 03:59:51.676533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.676036    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb" event={"ID":"29b368a6-341b-40aa-b072-aee1a1427533","Type":"ContainerStarted","Data":"a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15"}
Oct 31 03:59:51.676533 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.676160    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"]
Oct 31 03:59:51.679432 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.679399    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-ab21e4-7qt7d"]
Oct 31 03:59:51.935183 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.935155    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:59:51.956387 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.956318    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb" podStartSLOduration=1.956295043 podStartE2EDuration="1.956295043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 03:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 03:59:51.700139198 +0000 UTC m=+2666.225358788" watchObservedRunningTime="2026-10-31 03:59:51.956295043 +0000 UTC m=+2666.481514636"
Oct 31 03:59:51.975081 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:51.975045    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:59:52.095449 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.095407    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\" (UniqueName: \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\") pod \"40d382c5-1a49-46d8-93d1-2cacdae68564\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") "
Oct 31 03:59:52.095634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.095518    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\") pod \"40d382c5-1a49-46d8-93d1-2cacdae68564\" (UID: \"40d382c5-1a49-46d8-93d1-2cacdae68564\") "
Oct 31 03:59:52.095634 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.095568    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\") pod \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") "
Oct 31 03:59:52.095758 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.095645    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\" (UniqueName: \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\") pod \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\" (UID: \"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a\") "
Oct 31 03:59:52.095980 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.095951    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume" pod "40d382c5-1a49-46d8-93d1-2cacdae68564" (UID: "40d382c5-1a49-46d8-93d1-2cacdae68564"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:52.096123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.096030    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume" pod "60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" (UID: "60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:52.096195 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.096180    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40d382c5-1a49-46d8-93d1-2cacdae68564-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:52.096239 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.096198    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:52.097927 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.097895    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz" pod "40d382c5-1a49-46d8-93d1-2cacdae68564" (UID: "40d382c5-1a49-46d8-93d1-2cacdae68564"). InnerVolumeSpecName "kube-api-access-rnztz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:52.098050 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.097960    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4" pod "60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" (UID: "60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a"). InnerVolumeSpecName "kube-api-access-gscj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:52.196973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.196873    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnztz\" (UniqueName: \"kubernetes.io/projected/40d382c5-1a49-46d8-93d1-2cacdae68564-kube-api-access-rnztz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:52.196973 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.196908    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gscj4\" (UniqueName: \"kubernetes.io/projected/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a-kube-api-access-gscj4\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:52.685592 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.685555    5098 generic.go:356] "Generic (PLEG): container finished" podID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerID="8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9" exitCode=0
Oct 31 03:59:52.686106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.685669    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"
Oct 31 03:59:52.686106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.685674    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" event={"ID":"40d382c5-1a49-46d8-93d1-2cacdae68564","Type":"ContainerDied","Data":"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"}
Oct 31 03:59:52.686106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.685740    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" event={"ID":"40d382c5-1a49-46d8-93d1-2cacdae68564","Type":"ContainerDied","Data":"b257709a151dd919422863eecac68fa34daab08b41691e70237be546e0fe9ac0"}
Oct 31 03:59:52.686106 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.685770    5098 scope.go:122] "RemoveContainer" containerID="8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"
Oct 31 03:59:52.687431 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.687404    5098 generic.go:356] "Generic (PLEG): container finished" podID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerID="2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512" exitCode=0
Oct 31 03:59:52.687540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.687454    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"
Oct 31 03:59:52.687540 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.687488    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" event={"ID":"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a","Type":"ContainerDied","Data":"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"}
Oct 31 03:59:52.687715 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.687539    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" event={"ID":"60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a","Type":"ContainerDied","Data":"0ee73b7775c18f11f2e96961665618ce32f657e99aecb131574de8eee0591378"}
Oct 31 03:59:52.689783 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.689756    5098 generic.go:356] "Generic (PLEG): container finished" podID="29b368a6-341b-40aa-b072-aee1a1427533" containerID="1560671dca9a86100188de39923e351e96e9a0cc9a28fbd4c58fac392bed9f5b" exitCode=0
Oct 31 03:59:52.689921 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.689840    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb" event={"ID":"29b368a6-341b-40aa-b072-aee1a1427533","Type":"ContainerDied","Data":"1560671dca9a86100188de39923e351e96e9a0cc9a28fbd4c58fac392bed9f5b"}
Oct 31 03:59:52.697941 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.697921    5098 scope.go:122] "RemoveContainer" containerID="8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"
Oct 31 03:59:52.698310 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:52.698261    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9\": container with ID starting with 8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9 not found: ID does not exist" containerID="8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"
Oct 31 03:59:52.698420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.698321    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9"} err="failed to get container status \"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9\": rpc error: code = NotFound desc = could not find container \"8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9\": container with ID starting with 8a4a8e0cb3fe4150d3ade394415db914cbf40dea6dacb2bd9bdf13bcad8024d9 not found: ID does not exist"
Oct 31 03:59:52.698420 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.698344    5098 scope.go:122] "RemoveContainer" containerID="2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"
Oct 31 03:59:52.714863 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.714837    5098 scope.go:122] "RemoveContainer" containerID="2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"
Oct 31 03:59:52.715302 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:52.715274    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512\": container with ID starting with 2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512 not found: ID does not exist" containerID="2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"
Oct 31 03:59:52.715367 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.715317    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512"} err="failed to get container status \"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512\": rpc error: code = NotFound desc = could not find container \"2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512\": container with ID starting with 2d60d604c7dcdc6dabc618e467a61a9af64c5fe76dfb46ad3ff5148672a74512 not found: ID does not exist"
Oct 31 03:59:52.732196 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.732160    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:59:52.739328 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.739294    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg"]
Oct 31 03:59:52.750206 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.750172    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:59:52.755376 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.755344    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp"]
Oct 31 03:59:52.869970 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.869913    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-39e8cf-6d86d6454c-lnjzg" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.12:50052\" within 1s: context deadline exceeded"
Oct 31 03:59:52.870173 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:52.869920    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-39e8cf-7d7cdcd67-kc2rp" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.13:50052\" within 1s: context deadline exceeded"
Oct 31 03:59:52.870173 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:59:52.869994    5098 logging.go:55] [core] [Channel #8900 SubChannel #8902]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.13:50052", ServerName: "10.132.1.13:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.13:50052: operation was canceled"
Oct 31 03:59:52.870173 ip-10-0-129-132 kubenswrapper[5098]: W1031 03:59:52.869994    5098 logging.go:55] [core] [Channel #8899 SubChannel #8901]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.12:50052", ServerName: "10.132.1.12:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.12:50052: operation was canceled"
Oct 31 03:59:53.595194 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.595153    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing" containerID="cri-o://e41ddcfdb9b9aa77620d0a2354742f69060fc252a358b725cd3d34263ac3a457" gracePeriod=25
Oct 31 03:59:53.601663 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.601571    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing" containerID="cri-o://d82b37c183cdc2decb3f215c26b787f56f70525674b28eab02a0326817c16eca" gracePeriod=25
Oct 31 03:59:53.612911 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.612876    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" path="/var/lib/kubelet/pods/40d382c5-1a49-46d8-93d1-2cacdae68564/volumes"
Oct 31 03:59:53.613340 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.613325    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" path="/var/lib/kubelet/pods/60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a/volumes"
Oct 31 03:59:53.613763 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.613749    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52ac339-072a-4ef1-b39a-bfa4fe27a844" path="/var/lib/kubelet/pods/c52ac339-072a-4ef1-b39a-bfa4fe27a844/volumes"
Oct 31 03:59:53.855095 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:53.855024    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:54.017964 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.017925    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\" (UniqueName: \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\") pod \"29b368a6-341b-40aa-b072-aee1a1427533\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") "
Oct 31 03:59:54.018176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.017998    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\") pod \"29b368a6-341b-40aa-b072-aee1a1427533\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") "
Oct 31 03:59:54.018176 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.018135    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\") pod \"29b368a6-341b-40aa-b072-aee1a1427533\" (UID: \"29b368a6-341b-40aa-b072-aee1a1427533\") "
Oct 31 03:59:54.018391 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.018367    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca" pod "29b368a6-341b-40aa-b072-aee1a1427533" (UID: "29b368a6-341b-40aa-b072-aee1a1427533"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 03:59:54.018554 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.018534    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.020123 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.020091    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7" pod "29b368a6-341b-40aa-b072-aee1a1427533" (UID: "29b368a6-341b-40aa-b072-aee1a1427533"). InnerVolumeSpecName "kube-api-access-tpcb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:54.020299 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.020189    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca" pod "29b368a6-341b-40aa-b072-aee1a1427533" (UID: "29b368a6-341b-40aa-b072-aee1a1427533"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:54.119924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.119822    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29b368a6-341b-40aa-b072-aee1a1427533-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.119924 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.119861    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpcb7\" (UniqueName: \"kubernetes.io/projected/29b368a6-341b-40aa-b072-aee1a1427533-kube-api-access-tpcb7\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:54.595291 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.595251    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing" containerID="cri-o://44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a" gracePeriod=25
Oct 31 03:59:54.642185 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.642144    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 03:59:54.643216 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.643186    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 03:59:54.739134 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.739087    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb" event={"ID":"29b368a6-341b-40aa-b072-aee1a1427533","Type":"ContainerDied","Data":"a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15"}
Oct 31 03:59:54.739319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.739157    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4201043c30dc2d92c4db35c0e954de4f13b221fabe99ab319368255f3e4cb15"
Oct 31 03:59:54.739319 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:54.739233    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"
Oct 31 03:59:58.292688 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.292641    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main" containerID="cri-o://ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d" gracePeriod=45
Oct 31 03:59:58.554092 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.554023    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:59:58.672997 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.672964    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\" (UniqueName: \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.673208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673071    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.673208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673106    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.673208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673136    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.673208 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673179    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.674903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673257    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\") pod \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\" (UID: \"b8554b90-54d5-4f2a-aa1c-5c5d0644495b\") "
Oct 31 03:59:58.674903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673762    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.674903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.673997    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.674903 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.674160    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.676404 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.676374    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 03:59:58.677553 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.676867    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "kube-api-access-ssmrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 03:59:58.677839 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.677791    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm" pod "b8554b90-54d5-4f2a-aa1c-5c5d0644495b" (UID: "b8554b90-54d5-4f2a-aa1c-5c5d0644495b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 03:59:58.771268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.771228    5098 generic.go:356] "Generic (PLEG): container finished" podID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerID="ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d" exitCode=0
Oct 31 03:59:58.771268 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.771274    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"
Oct 31 03:59:58.771499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.771311    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" event={"ID":"b8554b90-54d5-4f2a-aa1c-5c5d0644495b","Type":"ContainerDied","Data":"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"}
Oct 31 03:59:58.771499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.771366    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk" event={"ID":"b8554b90-54d5-4f2a-aa1c-5c5d0644495b","Type":"ContainerDied","Data":"267b471cc56b9fffe9a811e5090f1d33b030f4adcf27ae1009db88e9639add1d"}
Oct 31 03:59:58.771499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.771384    5098 scope.go:122] "RemoveContainer" containerID="ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"
Oct 31 03:59:58.774499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774471    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.774499 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774500    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.774709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774516    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssmrw\" (UniqueName: \"kubernetes.io/projected/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-kube-api-access-ssmrw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.774709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774528    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.774709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774540    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.774709 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.774552    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8554b90-54d5-4f2a-aa1c-5c5d0644495b-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 03:59:58.785170 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.785145    5098 scope.go:122] "RemoveContainer" containerID="ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"
Oct 31 03:59:58.785542 ip-10-0-129-132 kubenswrapper[5098]: E1031 03:59:58.785515    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d\": container with ID starting with ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d not found: ID does not exist" containerID="ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"
Oct 31 03:59:58.785672 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.785551    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d"} err="failed to get container status \"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d\": rpc error: code = NotFound desc = could not find container \"ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d\": container with ID starting with ac0b7fe98961d3458b2ca0f8eabc28bc292cfd9955ae4ffdd09da9547068002d not found: ID does not exist"
Oct 31 03:59:58.797175 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.797143    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:59:58.802013 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:58.801984    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-7f3159/distinct-w6-7f3159-kserve-6f945dfdd8-wt8fk"]
Oct 31 03:59:59.611551 ip-10-0-129-132 kubenswrapper[5098]: I1031 03:59:59.611515    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" path="/var/lib/kubelet/pods/b8554b90-54d5-4f2a-aa1c-5c5d0644495b/volumes"
Oct 31 04:00:00.147582 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147215    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:00.147845 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147829    5098 cpu_manager.go:414] "containerMap: removing container" podUID="29b368a6-341b-40aa-b072-aee1a1427533" containerName="revoke-keys"
Oct 31 04:00:00.147899 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147847    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29b368a6-341b-40aa-b072-aee1a1427533" containerName="revoke-keys"
Oct 31 04:00:00.147899 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147865    5098 cpu_manager.go:414] "containerMap: removing container" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing"
Oct 31 04:00:00.147899 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147873    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing"
Oct 31 04:00:00.147999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147909    5098 cpu_manager.go:414] "containerMap: removing container" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing"
Oct 31 04:00:00.147999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147915    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing"
Oct 31 04:00:00.147999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147925    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main"
Oct 31 04:00:00.147999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.147929    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main"
Oct 31 04:00:00.148146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.148042    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="60bb8f8e-9ab4-46ca-a1a6-ddf0a8cdc01a" containerName="payload-processing"
Oct 31 04:00:00.148146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.148053    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8554b90-54d5-4f2a-aa1c-5c5d0644495b" containerName="main"
Oct 31 04:00:00.148146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.148063    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="29b368a6-341b-40aa-b072-aee1a1427533" containerName="revoke-keys"
Oct 31 04:00:00.148146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.148069    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="40d382c5-1a49-46d8-93d1-2cacdae68564" containerName="payload-pre-processing"
Oct 31 04:00:00.152882 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.152856    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:00.153002 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.152984    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.156883 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.156859    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.159889 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.159843    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:00:00.160171 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.160149    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:00:00.162359 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.162165    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:00:00.165632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.163281    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:00.167526 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.167500    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:00.290884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.290837    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.291119 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.291021    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzk5m\" (UniqueName: \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.291119 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.291071    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5nqj\" (UniqueName: \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.291215 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.291157    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.392203 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.392163    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.392431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.392290    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzk5m\" (UniqueName: \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.392431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.392324    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5nqj\" (UniqueName: \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.392431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.392362    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.393026 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.392998    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.393138 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.393094    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.409835 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.409748    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzk5m\" (UniqueName: \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\") pod \"maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.409835 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.409748    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5nqj\" (UniqueName: \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\") pod \"maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.474179 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.474136    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:00.482961 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.482933    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:00.770456 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.770364    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:00.787473 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.786771    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:00.797227 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:00.796852    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:01.061275 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:01.061017    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:00:01.812510 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.812470    5098 generic.go:356] "Generic (PLEG): container finished" podID="422ab807-b7c0-41fa-a619-111d4528ecc5" containerID="035dc5ce86dd20d97f652c93bbadde3d0c0331eb1484dda72e2046afb3f32b51" exitCode=6
Oct 31 04:00:01.813001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.812557    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns" event={"ID":"422ab807-b7c0-41fa-a619-111d4528ecc5","Type":"ContainerDied","Data":"035dc5ce86dd20d97f652c93bbadde3d0c0331eb1484dda72e2046afb3f32b51"}
Oct 31 04:00:01.813001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.812662    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns" event={"ID":"422ab807-b7c0-41fa-a619-111d4528ecc5","Type":"ContainerStarted","Data":"483f8e87e656ad0b82d08730b1041a64d1e30226b4e4c690e1be843c23a9e4c7"}
Oct 31 04:00:01.814990 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.814953    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerStarted","Data":"4c1ee084c3d57a9e9940775e1448d1607245e6ace0e056328d5ae53de5d6f98d"}
Oct 31 04:00:01.815113 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.815029    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerStarted","Data":"5742be5542df54f69e35cf0a019fb998b9e30eb55365fd7b14e01b78e9f3d30f"}
Oct 31 04:00:01.852435 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.852362    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" podStartSLOduration=1.852342511 podStartE2EDuration="1.852342511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:01.851679812 +0000 UTC m=+2676.376899397" watchObservedRunningTime="2026-10-31 04:00:01.852342511 +0000 UTC m=+2676.377562101"
Oct 31 04:00:01.985146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.985106    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:01.999125 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:01.999089    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:02.022592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.022056    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\") pod \"422ab807-b7c0-41fa-a619-111d4528ecc5\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") "
Oct 31 04:00:02.022592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.022226    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\" (UniqueName: \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\") pod \"422ab807-b7c0-41fa-a619-111d4528ecc5\" (UID: \"422ab807-b7c0-41fa-a619-111d4528ecc5\") "
Oct 31 04:00:02.025059 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.024950    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca" pod "422ab807-b7c0-41fa-a619-111d4528ecc5" (UID: "422ab807-b7c0-41fa-a619-111d4528ecc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:02.034585 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.034540    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj" pod "422ab807-b7c0-41fa-a619-111d4528ecc5" (UID: "422ab807-b7c0-41fa-a619-111d4528ecc5"). InnerVolumeSpecName "kube-api-access-x5nqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:02.123515 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.123469    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5nqj\" (UniqueName: \"kubernetes.io/projected/422ab807-b7c0-41fa-a619-111d4528ecc5-kube-api-access-x5nqj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:02.123515 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.123508    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/422ab807-b7c0-41fa-a619-111d4528ecc5-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:02.824126 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.824083    5098 generic.go:356] "Generic (PLEG): container finished" podID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerID="4c1ee084c3d57a9e9940775e1448d1607245e6ace0e056328d5ae53de5d6f98d" exitCode=60
Oct 31 04:00:02.824578 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.824159    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerDied","Data":"4c1ee084c3d57a9e9940775e1448d1607245e6ace0e056328d5ae53de5d6f98d"}
Oct 31 04:00:02.824578 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.824483    5098 scope.go:122] "RemoveContainer" containerID="4c1ee084c3d57a9e9940775e1448d1607245e6ace0e056328d5ae53de5d6f98d"
Oct 31 04:00:02.825838 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.825812    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns" event={"ID":"422ab807-b7c0-41fa-a619-111d4528ecc5","Type":"ContainerDied","Data":"483f8e87e656ad0b82d08730b1041a64d1e30226b4e4c690e1be843c23a9e4c7"}
Oct 31 04:00:02.825965 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.825841    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"
Oct 31 04:00:02.825965 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.825861    5098 scope.go:122] "RemoveContainer" containerID="035dc5ce86dd20d97f652c93bbadde3d0c0331eb1484dda72e2046afb3f32b51"
Oct 31 04:00:02.861105 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.861068    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:02.864207 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:02.864171    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7f3159-29890320-kdrns"]
Oct 31 04:00:03.006330 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.006281    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:03.615773 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.615734    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422ab807-b7c0-41fa-a619-111d4528ecc5" path="/var/lib/kubelet/pods/422ab807-b7c0-41fa-a619-111d4528ecc5/volumes"
Oct 31 04:00:03.835313 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.835273    5098 generic.go:356] "Generic (PLEG): container finished" podID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerID="e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118" exitCode=60
Oct 31 04:00:03.835852 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.835323    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerDied","Data":"e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118"}
Oct 31 04:00:03.835852 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.835391    5098 scope.go:122] "RemoveContainer" containerID="4c1ee084c3d57a9e9940775e1448d1607245e6ace0e056328d5ae53de5d6f98d"
Oct 31 04:00:03.835852 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:03.835694    5098 scope.go:122] "RemoveContainer" containerID="e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118"
Oct 31 04:00:03.836072 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:03.836043    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd_odh-ai-gateway-infra(bb66b743-2410-4f2c-8f5f-1334d8759d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08"
Oct 31 04:00:04.410881 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:04.410843    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 04:00:04.416020 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:04.415980    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 04:00:04.644016 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:04.643977    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:04.847112 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:04.847084    5098 scope.go:122] "RemoveContainer" containerID="e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118"
Oct 31 04:00:04.847685 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:04.847350    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd_odh-ai-gateway-infra(bb66b743-2410-4f2c-8f5f-1334d8759d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08"
Oct 31 04:00:08.826147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:08.826104    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 04:00:08.826756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:08.826370    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" containerID="cri-o://c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3" gracePeriod=30
Oct 31 04:00:09.328083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:09.328046    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:00:09.820352 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:09.820305    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 04:00:10.532649 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.532594    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"]
Oct 31 04:00:10.533159 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.533145    5098 cpu_manager.go:414] "containerMap: removing container" podUID="422ab807-b7c0-41fa-a619-111d4528ecc5" containerName="cleanup"
Oct 31 04:00:10.533222 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.533160    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422ab807-b7c0-41fa-a619-111d4528ecc5" containerName="cleanup"
Oct 31 04:00:10.533262 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.533242    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="422ab807-b7c0-41fa-a619-111d4528ecc5" containerName="cleanup"
Oct 31 04:00:10.538243 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.537808    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.543479 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.543450    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:00:10.545938 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.545908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"]
Oct 31 04:00:10.634770 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.634713    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.634970 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.634843    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmzlq\" (UniqueName: \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.634970 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.634895    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.736157 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.736106    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.736399 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.736184    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmzlq\" (UniqueName: \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.736399 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.736216    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.736924 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.736899    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.738717 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.738687    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.744578 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.744547    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmzlq\" (UniqueName: \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\") pod \"maas-api-revoke-sub-8e39eef6-rz2x8\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:10.864049 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:10.863944    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:11.153168 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:11.151059    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"]
Oct 31 04:00:11.901977 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:11.901937    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8" event={"ID":"29257438-7afd-4910-b02f-2a4581705f7d","Type":"ContainerStarted","Data":"1fba6328b12b853a3c499b2a5701cdfed007fc11121165b4e1e09cbb735f1637"}
Oct 31 04:00:11.902399 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:11.901997    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8" event={"ID":"29257438-7afd-4910-b02f-2a4581705f7d","Type":"ContainerStarted","Data":"3e9bafdcf134474b4dad16917de9df31a709e7488a00d6f32d2126f67e628bbb"}
Oct 31 04:00:11.919917 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:11.919854    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8" podStartSLOduration=1.9198413319999998 podStartE2EDuration="1.919841332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:00:11.919267349 +0000 UTC m=+2686.444486951" watchObservedRunningTime="2026-10-31 04:00:11.919841332 +0000 UTC m=+2686.445060977"
Oct 31 04:00:11.999141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:11.999099    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:12.911204 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:12.911166    5098 generic.go:356] "Generic (PLEG): container finished" podID="29257438-7afd-4910-b02f-2a4581705f7d" containerID="1fba6328b12b853a3c499b2a5701cdfed007fc11121165b4e1e09cbb735f1637" exitCode=0
Oct 31 04:00:12.911698 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:12.911252    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8" event={"ID":"29257438-7afd-4910-b02f-2a4581705f7d","Type":"ContainerDied","Data":"1fba6328b12b853a3c499b2a5701cdfed007fc11121165b4e1e09cbb735f1637"}
Oct 31 04:00:13.005991 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.005949    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:13.920798 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.920762    5098 generic.go:356] "Generic (PLEG): container finished" podID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerID="e41ddcfdb9b9aa77620d0a2354742f69060fc252a358b725cd3d34263ac3a457" exitCode=0
Oct 31 04:00:13.921302 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.920833    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" event={"ID":"fc210d02-f4a2-4f76-8a6f-12484b05d3ac","Type":"ContainerDied","Data":"e41ddcfdb9b9aa77620d0a2354742f69060fc252a358b725cd3d34263ac3a457"}
Oct 31 04:00:13.922640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.922590    5098 generic.go:356] "Generic (PLEG): container finished" podID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerID="d82b37c183cdc2decb3f215c26b787f56f70525674b28eab02a0326817c16eca" exitCode=0
Oct 31 04:00:13.922750 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.922635    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" event={"ID":"cab7e371-cb9c-46ca-be9c-faa8000ba33f","Type":"ContainerDied","Data":"d82b37c183cdc2decb3f215c26b787f56f70525674b28eab02a0326817c16eca"}
Oct 31 04:00:13.945061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.945035    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 04:00:13.949842 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:13.949809    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 04:00:14.076241 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076189    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\") pod \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") "
Oct 31 04:00:14.076452 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076307    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\") pod \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") "
Oct 31 04:00:14.076452 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076370    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\" (UniqueName: \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\") pod \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\" (UID: \"cab7e371-cb9c-46ca-be9c-faa8000ba33f\") "
Oct 31 04:00:14.076592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076518    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\" (UniqueName: \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\") pod \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\" (UID: \"fc210d02-f4a2-4f76-8a6f-12484b05d3ac\") "
Oct 31 04:00:14.077056 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076792    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume" pod "fc210d02-f4a2-4f76-8a6f-12484b05d3ac" (UID: "fc210d02-f4a2-4f76-8a6f-12484b05d3ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:14.077056 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.076861    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume" pod "cab7e371-cb9c-46ca-be9c-faa8000ba33f" (UID: "cab7e371-cb9c-46ca-be9c-faa8000ba33f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:14.077056 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.077035    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.077056 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.077057    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cab7e371-cb9c-46ca-be9c-faa8000ba33f-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.079216 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.079189    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn" pod "fc210d02-f4a2-4f76-8a6f-12484b05d3ac" (UID: "fc210d02-f4a2-4f76-8a6f-12484b05d3ac"). InnerVolumeSpecName "kube-api-access-hknrn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.079332 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.079255    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr" pod "cab7e371-cb9c-46ca-be9c-faa8000ba33f" (UID: "cab7e371-cb9c-46ca-be9c-faa8000ba33f"). InnerVolumeSpecName "kube-api-access-bk5dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.093968 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.093941    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:14.096751 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.096727    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 04:00:14.177939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.177829    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.177939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.177895    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\") pod \"29257438-7afd-4910-b02f-2a4581705f7d\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") "
Oct 31 04:00:14.177939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.177940    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.177983    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178031    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178071    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178096    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178190    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\") pod \"29257438-7afd-4910-b02f-2a4581705f7d\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178213    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178245    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178273    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\" (UniqueName: \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\") pod \"29257438-7afd-4910-b02f-2a4581705f7d\" (UID: \"29257438-7afd-4910-b02f-2a4581705f7d\") "
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178289    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178312    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\") pod \"c046f55b-2491-441e-9099-f9df90217489\" (UID: \"c046f55b-2491-441e-9099-f9df90217489\") "
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178311    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca" pod "29257438-7afd-4910-b02f-2a4581705f7d" (UID: "29257438-7afd-4910-b02f-2a4581705f7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178589    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:14.178682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178657    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178810    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c046f55b-2491-441e-9099-f9df90217489-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178829    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178848    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hknrn\" (UniqueName: \"kubernetes.io/projected/fc210d02-f4a2-4f76-8a6f-12484b05d3ac-kube-api-access-hknrn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178863    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178881    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk5dr\" (UniqueName: \"kubernetes.io/projected/cab7e371-cb9c-46ca-be9c-faa8000ba33f-kube-api-access-bk5dr\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.178896    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.179346 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.179318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:14.179410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.179379    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:14.180709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.180680    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:00:14.180908 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.180887    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.181156 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.181128    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca" pod "29257438-7afd-4910-b02f-2a4581705f7d" (UID: "29257438-7afd-4910-b02f-2a4581705f7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:00:14.181256 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.181171    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq" pod "29257438-7afd-4910-b02f-2a4581705f7d" (UID: "29257438-7afd-4910-b02f-2a4581705f7d"). InnerVolumeSpecName "kube-api-access-rmzlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.181305 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.181257    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "kube-api-access-r82mm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.181374 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.181346    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy" pod "c046f55b-2491-441e-9099-f9df90217489" (UID: "c046f55b-2491-441e-9099-f9df90217489"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:14.280149 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280111    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r82mm\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-kube-api-access-r82mm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280149 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280141    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c046f55b-2491-441e-9099-f9df90217489-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280149 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280154    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c046f55b-2491-441e-9099-f9df90217489-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280163    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280174    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29257438-7afd-4910-b02f-2a4581705f7d-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280189    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280202    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c046f55b-2491-441e-9099-f9df90217489-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.280487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.280213    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmzlq\" (UniqueName: \"kubernetes.io/projected/29257438-7afd-4910-b02f-2a4581705f7d-kube-api-access-rmzlq\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:14.642923 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:00:14.642893    5098 logging.go:55] [core] [Channel #8933 SubChannel #8934]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.166:50052", ServerName: "10.132.0.166:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.166:50052: connect: connection refused"
Oct 31 04:00:14.820842 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.820748    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing" containerID="cri-o://7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d" gracePeriod=25
Oct 31 04:00:14.873197 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.873165    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 04:00:14.931418 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.931384    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"
Oct 31 04:00:14.931418 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.931392    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8" event={"ID":"29257438-7afd-4910-b02f-2a4581705f7d","Type":"ContainerDied","Data":"3e9bafdcf134474b4dad16917de9df31a709e7488a00d6f32d2126f67e628bbb"}
Oct 31 04:00:14.931950 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.931448    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e9bafdcf134474b4dad16917de9df31a709e7488a00d6f32d2126f67e628bbb"
Oct 31 04:00:14.932999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.932973    5098 generic.go:356] "Generic (PLEG): container finished" podID="c046f55b-2491-441e-9099-f9df90217489" containerID="c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3" exitCode=0
Oct 31 04:00:14.933124 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.933029    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"
Oct 31 04:00:14.933124 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.933071    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" event={"ID":"c046f55b-2491-441e-9099-f9df90217489","Type":"ContainerDied","Data":"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"}
Oct 31 04:00:14.933124 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.933118    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" event={"ID":"c046f55b-2491-441e-9099-f9df90217489","Type":"ContainerDied","Data":"c86055de9cbe0ea6d109cc45f32a793bf54c542cde37d2de5d9a2885366e8744"}
Oct 31 04:00:14.933278 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.933142    5098 scope.go:122] "RemoveContainer" containerID="c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"
Oct 31 04:00:14.934818 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.934799    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"
Oct 31 04:00:14.934934 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.934777    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6" event={"ID":"fc210d02-f4a2-4f76-8a6f-12484b05d3ac","Type":"ContainerDied","Data":"0cefbe77bf926fb2c19f7bcb5ca5fc2c9eff6a84498906680b994a0bcc6e0a37"}
Oct 31 04:00:14.942350 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.936649    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"
Oct 31 04:00:14.942350 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.936665    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7" event={"ID":"cab7e371-cb9c-46ca-be9c-faa8000ba33f","Type":"ContainerDied","Data":"4ad73de03959d53fb8a3f7cc08fcef76426a9a8235b89f51ad05c00757b9e5b2"}
Oct 31 04:00:14.942884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.942850    5098 generic.go:356] "Generic (PLEG): container finished" podID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerID="44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a" exitCode=0
Oct 31 04:00:14.943022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.942885    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" event={"ID":"3f9979b4-4472-4277-96cc-b5f430090bb2","Type":"ContainerDied","Data":"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"}
Oct 31 04:00:14.943022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.942892    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"
Oct 31 04:00:14.943022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.942927    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" event={"ID":"3f9979b4-4472-4277-96cc-b5f430090bb2","Type":"ContainerDied","Data":"2844d625d57d7e1b581906f8357f2f74266a9ca23e919c1c2d48417d48627917"}
Oct 31 04:00:14.947078 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.947057    5098 scope.go:122] "RemoveContainer" containerID="c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"
Oct 31 04:00:14.947391 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:14.947371    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3\": container with ID starting with c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3 not found: ID does not exist" containerID="c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"
Oct 31 04:00:14.947481 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.947397    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3"} err="failed to get container status \"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3\": rpc error: code = NotFound desc = could not find container \"c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3\": container with ID starting with c5aab7fb1cbb8a1c4a76c2c8fc8addf24fb53218c59f6e0bf793bee24c5a27b3 not found: ID does not exist"
Oct 31 04:00:14.947481 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.947412    5098 scope.go:122] "RemoveContainer" containerID="e41ddcfdb9b9aa77620d0a2354742f69060fc252a358b725cd3d34263ac3a457"
Oct 31 04:00:14.958255 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.958226    5098 scope.go:122] "RemoveContainer" containerID="d82b37c183cdc2decb3f215c26b787f56f70525674b28eab02a0326817c16eca"
Oct 31 04:00:14.969248 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.969086    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 04:00:14.969325 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.969274    5098 scope.go:122] "RemoveContainer" containerID="44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"
Oct 31 04:00:14.979823 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.979793    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m"]
Oct 31 04:00:14.981399 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.981379    5098 scope.go:122] "RemoveContainer" containerID="44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"
Oct 31 04:00:14.981753 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:14.981731    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a\": container with ID starting with 44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a not found: ID does not exist" containerID="44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"
Oct 31 04:00:14.981831 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.981767    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a"} err="failed to get container status \"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a\": rpc error: code = NotFound desc = could not find container \"44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a\": container with ID starting with 44a5371b8f91ca2bf867b9a43b2ea3b48aabee45c3c49ee780047a397b78df9a not found: ID does not exist"
Oct 31 04:00:14.987337 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.987315    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\") pod \"3f9979b4-4472-4277-96cc-b5f430090bb2\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") "
Oct 31 04:00:14.987442 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.987363    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\" (UniqueName: \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\") pod \"3f9979b4-4472-4277-96cc-b5f430090bb2\" (UID: \"3f9979b4-4472-4277-96cc-b5f430090bb2\") "
Oct 31 04:00:14.987710 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.987687    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume" pod "3f9979b4-4472-4277-96cc-b5f430090bb2" (UID: "3f9979b4-4472-4277-96cc-b5f430090bb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:14.989533 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.989500    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f" pod "3f9979b4-4472-4277-96cc-b5f430090bb2" (UID: "3f9979b4-4472-4277-96cc-b5f430090bb2"). InnerVolumeSpecName "kube-api-access-m8w5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:14.994161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.994135    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 04:00:14.998595 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:14.998571    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-7f3159-577f89cd87-gkzl7"]
Oct 31 04:00:15.009617 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.009582    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 04:00:15.013431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.013406    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-7f3159-598b89d49-hgwv6"]
Oct 31 04:00:15.048736 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.048697    5098 patch_prober.go:29] interesting pod/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:00:15.048872 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.048765    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7f3159-openshift-default-5f9657f95-k556m" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:00:15.089133 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.089095    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f9979b4-4472-4277-96cc-b5f430090bb2-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:15.089133 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.089125    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8w5f\" (UniqueName: \"kubernetes.io/projected/3f9979b4-4472-4277-96cc-b5f430090bb2-kube-api-access-m8w5f\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:15.266928 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.266892    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 04:00:15.271655 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.271592    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd"]
Oct 31 04:00:15.611116 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.611076    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" path="/var/lib/kubelet/pods/3f9979b4-4472-4277-96cc-b5f430090bb2/volumes"
Oct 31 04:00:15.611556 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.611542    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c046f55b-2491-441e-9099-f9df90217489" path="/var/lib/kubelet/pods/c046f55b-2491-441e-9099-f9df90217489/volumes"
Oct 31 04:00:15.611970 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.611955    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" path="/var/lib/kubelet/pods/cab7e371-cb9c-46ca-be9c-faa8000ba33f/volumes"
Oct 31 04:00:15.612616 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.612590    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" path="/var/lib/kubelet/pods/fc210d02-f4a2-4f76-8a6f-12484b05d3ac/volumes"
Oct 31 04:00:15.642193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:15.642145    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w5-ab21e4-7ccdd69c87-q5hcd" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.166:50052\" within 1s: context deadline exceeded"
Oct 31 04:00:18.604435 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:18.604397    5098 scope.go:122] "RemoveContainer" containerID="e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118"
Oct 31 04:00:18.985713 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:18.985674    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerStarted","Data":"41d20c28266446f1b021462e3aa05a55f79bbd25588edc09a2078c79d155bda2"}
Oct 31 04:00:19.433262 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.433193    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main" containerID="cri-o://fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54" gracePeriod=45
Oct 31 04:00:19.437988 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.437940    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" containerID="cri-o://2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821" gracePeriod=45
Oct 31 04:00:19.717141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.717109    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 04:00:19.759876 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.759843    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 04:00:19.841508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841474    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\" (UniqueName: \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841527    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841566    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841633    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\" (UniqueName: \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841680    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841699    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\" (UniqueName: \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841737    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841763    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841788    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841802    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841863    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\" (UniqueName: \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\") pod \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\" (UID: \"510f30c0-9f70-4c49-b0c1-96fa9586aee8\") "
Oct 31 04:00:19.842042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.841926    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\") pod \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\" (UID: \"69fad609-0f14-4d2c-8249-6e96a8ec3d30\") "
Oct 31 04:00:19.842904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.842754    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.843009 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.842971    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.843146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.843121    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.843252 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.843194    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.843401 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.843318    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.843466 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.843455    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.845250 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.844564    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.846146 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.846058    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:19.846262 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.846166    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:00:19.846524 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.846495    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj" pod "510f30c0-9f70-4c49-b0c1-96fa9586aee8" (UID: "510f30c0-9f70-4c49-b0c1-96fa9586aee8"). InnerVolumeSpecName "kube-api-access-jxhnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.846650 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.846549    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "kube-api-access-zwkcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:19.846650 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.846565    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs" pod "69fad609-0f14-4d2c-8249-6e96a8ec3d30" (UID: "69fad609-0f14-4d2c-8249-6e96a8ec3d30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943795    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943834    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwkcv\" (UniqueName: \"kubernetes.io/projected/69fad609-0f14-4d2c-8249-6e96a8ec3d30-kube-api-access-zwkcv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943848    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943860    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943875    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxhnj\" (UniqueName: \"kubernetes.io/projected/510f30c0-9f70-4c49-b0c1-96fa9586aee8-kube-api-access-jxhnj\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943887    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.943904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943901    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.944294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943912    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.944294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943926    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.944294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943939    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69fad609-0f14-4d2c-8249-6e96a8ec3d30-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.944294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943950    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/510f30c0-9f70-4c49-b0c1-96fa9586aee8-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.944294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.943962    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/510f30c0-9f70-4c49-b0c1-96fa9586aee8-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:19.995634 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.995573    5098 generic.go:356] "Generic (PLEG): container finished" podID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerID="2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821" exitCode=0
Oct 31 04:00:19.995841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.995672    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"
Oct 31 04:00:19.995841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.995672    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" event={"ID":"510f30c0-9f70-4c49-b0c1-96fa9586aee8","Type":"ContainerDied","Data":"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"}
Oct 31 04:00:19.995841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.995732    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" event={"ID":"510f30c0-9f70-4c49-b0c1-96fa9586aee8","Type":"ContainerDied","Data":"7f5d24c2628b62230e168959cffb5fcdc5c5e1ff99f9cb3b5be34006b9f7b249"}
Oct 31 04:00:19.995841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.995754    5098 scope.go:122] "RemoveContainer" containerID="2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"
Oct 31 04:00:19.997345 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.997319    5098 generic.go:356] "Generic (PLEG): container finished" podID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerID="fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54" exitCode=0
Oct 31 04:00:19.997489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.997363    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"
Oct 31 04:00:19.997489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.997404    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" event={"ID":"69fad609-0f14-4d2c-8249-6e96a8ec3d30","Type":"ContainerDied","Data":"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"}
Oct 31 04:00:19.997489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.997462    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58" event={"ID":"69fad609-0f14-4d2c-8249-6e96a8ec3d30","Type":"ContainerDied","Data":"54df182afaff43c9caf0775e8186ef05520a61a0ae2709d1fb4f75676a73537d"}
Oct 31 04:00:19.999949 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.999924    5098 generic.go:356] "Generic (PLEG): container finished" podID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerID="41d20c28266446f1b021462e3aa05a55f79bbd25588edc09a2078c79d155bda2" exitCode=60
Oct 31 04:00:20.000125 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:19.999984    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerDied","Data":"41d20c28266446f1b021462e3aa05a55f79bbd25588edc09a2078c79d155bda2"}
Oct 31 04:00:20.000397 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.000383    5098 scope.go:122] "RemoveContainer" containerID="41d20c28266446f1b021462e3aa05a55f79bbd25588edc09a2078c79d155bda2"
Oct 31 04:00:20.000727 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:20.000704    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd_odh-ai-gateway-infra(bb66b743-2410-4f2c-8f5f-1334d8759d08)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08"
Oct 31 04:00:20.009904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.009884    5098 scope.go:122] "RemoveContainer" containerID="2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"
Oct 31 04:00:20.010242 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:20.010219    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821\": container with ID starting with 2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821 not found: ID does not exist" containerID="2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"
Oct 31 04:00:20.010315 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.010256    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821"} err="failed to get container status \"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821\": rpc error: code = NotFound desc = could not find container \"2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821\": container with ID starting with 2d18200e5af34b3845da043914547b06f048d16e5663af59d4e964e3fee03821 not found: ID does not exist"
Oct 31 04:00:20.010315 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.010284    5098 scope.go:122] "RemoveContainer" containerID="fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"
Oct 31 04:00:20.021337 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.021212    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:20.021481 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.021372    5098 scope.go:122] "RemoveContainer" containerID="fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"
Oct 31 04:00:20.022435 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:20.022409    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54\": container with ID starting with fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54 not found: ID does not exist" containerID="fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"
Oct 31 04:00:20.022582 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.022444    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54"} err="failed to get container status \"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54\": rpc error: code = NotFound desc = could not find container \"fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54\": container with ID starting with fd09dd12044f46d211c77a28881bf3171052b3b73ccfaefc90835f19472aeb54 not found: ID does not exist"
Oct 31 04:00:20.022582 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.022470    5098 scope.go:122] "RemoveContainer" containerID="e9115e1646b561381564e3a240efb041a6b2171d725f317c676210b9ff126118"
Oct 31 04:00:20.058825 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.058780    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 04:00:20.063536 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.063507    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-2-w5-bcc7a4-kserve-57cdb5777f-l9f58"]
Oct 31 04:00:20.075322 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.075288    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 04:00:20.079487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.079450    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb"]
Oct 31 04:00:20.659392 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.659345    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-bcc7a4/distinct-w5-bcc7a4-kserve-79b87c7764-cbqxb" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:00:20.987207 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:20.987118    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:21.176263 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.176236    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:21.256981 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.256871    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\") pod \"bb66b743-2410-4f2c-8f5f-1334d8759d08\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") "
Oct 31 04:00:21.256981 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.256963    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\" (UniqueName: \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\") pod \"bb66b743-2410-4f2c-8f5f-1334d8759d08\" (UID: \"bb66b743-2410-4f2c-8f5f-1334d8759d08\") "
Oct 31 04:00:21.259655 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.258128    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca" pod "bb66b743-2410-4f2c-8f5f-1334d8759d08" (UID: "bb66b743-2410-4f2c-8f5f-1334d8759d08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:21.260193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.260159    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m" pod "bb66b743-2410-4f2c-8f5f-1334d8759d08" (UID: "bb66b743-2410-4f2c-8f5f-1334d8759d08"). InnerVolumeSpecName "kube-api-access-pzk5m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:21.358193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.358152    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb66b743-2410-4f2c-8f5f-1334d8759d08-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:21.358193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.358186    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzk5m\" (UniqueName: \"kubernetes.io/projected/bb66b743-2410-4f2c-8f5f-1334d8759d08-kube-api-access-pzk5m\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:21.611206 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.611171    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" path="/var/lib/kubelet/pods/510f30c0-9f70-4c49-b0c1-96fa9586aee8/volumes"
Oct 31 04:00:21.611639 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:21.611621    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" path="/var/lib/kubelet/pods/69fad609-0f14-4d2c-8249-6e96a8ec3d30/volumes"
Oct 31 04:00:22.021931 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:22.021847    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"
Oct 31 04:00:22.021931 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:22.021876    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd" event={"ID":"bb66b743-2410-4f2c-8f5f-1334d8759d08","Type":"ContainerDied","Data":"5742be5542df54f69e35cf0a019fb998b9e30eb55365fd7b14e01b78e9f3d30f"}
Oct 31 04:00:22.022525 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:22.021959    5098 scope.go:122] "RemoveContainer" containerID="41d20c28266446f1b021462e3aa05a55f79bbd25588edc09a2078c79d155bda2"
Oct 31 04:00:22.041329 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:22.041289    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:22.044677 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:22.044645    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-bcc7a4-29890320-n7fzd"]
Oct 31 04:00:23.033049 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:23.033017    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:00:23.612173 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:23.612129    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" path="/var/lib/kubelet/pods/bb66b743-2410-4f2c-8f5f-1334d8759d08/volumes"
Oct 31 04:00:30.772631 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:30.772575    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:00:30.987154 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:30.987106    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:00:31.057688 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:31.057577    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:00:32.390507 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.390466    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 04:00:32.391055 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.390782    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy" containerID="cri-o://3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0" gracePeriod=30
Oct 31 04:00:32.394457 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.394425    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 04:00:32.394720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.394689    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy" containerID="cri-o://1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7" gracePeriod=30
Oct 31 04:00:32.953894 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.953863    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:00:32.972744 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:32.972713    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:00:35.094569 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.094499    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 04:00:35.101904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.101871    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\") pod \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") "
Oct 31 04:00:35.102181 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.101955    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\") pod \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\" (UID: \"8a4810ca-5db9-4232-a5fe-1abf2b78cc54\") "
Oct 31 04:00:35.102309 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.102285    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume" pod "8a4810ca-5db9-4232-a5fe-1abf2b78cc54" (UID: "8a4810ca-5db9-4232-a5fe-1abf2b78cc54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:35.102423 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.102407    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:35.104347 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.104317    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9" pod "8a4810ca-5db9-4232-a5fe-1abf2b78cc54" (UID: "8a4810ca-5db9-4232-a5fe-1abf2b78cc54"). InnerVolumeSpecName "kube-api-access-j9bn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:35.124376 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.124336    5098 generic.go:356] "Generic (PLEG): container finished" podID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerID="7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d" exitCode=0
Oct 31 04:00:35.124592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.124391    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" event={"ID":"8a4810ca-5db9-4232-a5fe-1abf2b78cc54","Type":"ContainerDied","Data":"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"}
Oct 31 04:00:35.124592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.124455    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv" event={"ID":"8a4810ca-5db9-4232-a5fe-1abf2b78cc54","Type":"ContainerDied","Data":"8622b93fb7df721c23584ad4805decafb76cc284f0b33c0182187ba54f6b6c09"}
Oct 31 04:00:35.124592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.124400    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"
Oct 31 04:00:35.124592 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.124479    5098 scope.go:122] "RemoveContainer" containerID="7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"
Oct 31 04:00:35.136738 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.136716    5098 scope.go:122] "RemoveContainer" containerID="7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"
Oct 31 04:00:35.137083 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:35.137054    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d\": container with ID starting with 7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d not found: ID does not exist" containerID="7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"
Oct 31 04:00:35.137210 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.137097    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d"} err="failed to get container status \"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d\": rpc error: code = NotFound desc = could not find container \"7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d\": container with ID starting with 7e68f7fe812077621ef4861db523d1aac05c7709784d491def11ea009501486d not found: ID does not exist"
Oct 31 04:00:35.151881 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.151843    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 04:00:35.157389 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.157355    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-bcc7a4-5b5fc659dc-vv4rv"]
Oct 31 04:00:35.203816 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.203779    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9bn9\" (UniqueName: \"kubernetes.io/projected/8a4810ca-5db9-4232-a5fe-1abf2b78cc54-kube-api-access-j9bn9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:35.610117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:35.610081    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" path="/var/lib/kubelet/pods/8a4810ca-5db9-4232-a5fe-1abf2b78cc54/volumes"
Oct 31 04:00:37.697674 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.697639    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 04:00:37.729575 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729538    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729575 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729577    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729634    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729684    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729741    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729772    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.729848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729807    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.730141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729922    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.730141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729903    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.730141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729967    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\") pod \"d6c3302e-f961-43a9-b381-a9979bca3d1f\" (UID: \"d6c3302e-f961-43a9-b381-a9979bca3d1f\") "
Oct 31 04:00:37.730141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.729991    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.730438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730144    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.730438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730349    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.730438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730410    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.730438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730426    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.730694 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730443    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.730891 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.730861    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:37.732377 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.732349    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:37.732806 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.732779    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.732909 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.732819    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "kube-api-access-hdv6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:37.732909 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.732830    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo" pod "d6c3302e-f961-43a9-b381-a9979bca3d1f" (UID: "d6c3302e-f961-43a9-b381-a9979bca3d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:00:37.733558 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.733540    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 04:00:37.831787 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.831739    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.831787 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.831787    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.831817    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.831856    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832021    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832056    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832092    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832151    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832175    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.832285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832196    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\") pod \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\" (UID: \"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882\") "
Oct 31 04:00:37.832517 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832314    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:00:37.832590 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832519    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.832590 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832536    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.832735 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832466    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.832797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832744    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdv6w\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-kube-api-access-hdv6w\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832764    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6c3302e-f961-43a9-b381-a9979bca3d1f-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832782    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832796    5098 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-data\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832810    5098 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832824    5098 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-workload-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832839    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832853    5098 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-credential-socket\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832868    5098 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istiod-ca-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832882    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.832962 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.832895    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6c3302e-f961-43a9-b381-a9979bca3d1f-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.834297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.834264    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:00:37.834413 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.834366    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:37.834583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.834563    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 04:00:37.834583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.834568    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8" pod "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" (UID: "dfa609e7-8ef4-41d3-a2da-3b7bf9dec882"). InnerVolumeSpecName "kube-api-access-4vlh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:00:37.934147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.934058    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vlh8\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-kube-api-access-4vlh8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.934147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.934099    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.934147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.934111    5098 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-envoy\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:37.934147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:37.934120    5098 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882-istio-podinfo\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:00:38.154058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.154012    5098 generic.go:356] "Generic (PLEG): container finished" podID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerID="3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0" exitCode=0
Oct 31 04:00:38.154266 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.154099    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"
Oct 31 04:00:38.154266 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.154092    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerDied","Data":"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"}
Oct 31 04:00:38.154266 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.154256    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr" event={"ID":"dfa609e7-8ef4-41d3-a2da-3b7bf9dec882","Type":"ContainerDied","Data":"336fc32b3cbdcdc18e6866c7ba14c325049c326ad702da9d1f9a6cf40712c41a"}
Oct 31 04:00:38.154412 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.154295    5098 scope.go:122] "RemoveContainer" containerID="3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"
Oct 31 04:00:38.161455 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.156082    5098 generic.go:356] "Generic (PLEG): container finished" podID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerID="1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7" exitCode=0
Oct 31 04:00:38.161455 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.156128    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" event={"ID":"d6c3302e-f961-43a9-b381-a9979bca3d1f","Type":"ContainerDied","Data":"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"}
Oct 31 04:00:38.161455 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.156170    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz" event={"ID":"d6c3302e-f961-43a9-b381-a9979bca3d1f","Type":"ContainerDied","Data":"6aa04d70d6d7477356f84902928513e378a883d08ee52f764e91eede8a6b628e"}
Oct 31 04:00:38.161455 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.156133    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"
Oct 31 04:00:38.170444 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.170385    5098 scope.go:122] "RemoveContainer" containerID="e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"
Oct 31 04:00:38.189293 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.189142    5098 scope.go:122] "RemoveContainer" containerID="3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"
Oct 31 04:00:38.189670 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:38.189637    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0\": container with ID starting with 3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0 not found: ID does not exist" containerID="3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"
Oct 31 04:00:38.189795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.189682    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0"} err="failed to get container status \"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0\": rpc error: code = NotFound desc = could not find container \"3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0\": container with ID starting with 3874eded6c461adf79c28560624ab01b7ce93a02da6bc543f4fac82c061481b0 not found: ID does not exist"
Oct 31 04:00:38.189795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.189707    5098 scope.go:122] "RemoveContainer" containerID="e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"
Oct 31 04:00:38.190063 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:38.190037    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740\": container with ID starting with e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740 not found: ID does not exist" containerID="e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"
Oct 31 04:00:38.190156 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.190071    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740"} err="failed to get container status \"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740\": rpc error: code = NotFound desc = could not find container \"e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740\": container with ID starting with e60fea18b254af7cdedd4ccad647da7850f1d16faf6dcb3c416e3d91c7fa2740 not found: ID does not exist"
Oct 31 04:00:38.190156 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.190092    5098 scope.go:122] "RemoveContainer" containerID="1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"
Oct 31 04:00:38.205846 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.205813    5098 scope.go:122] "RemoveContainer" containerID="1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"
Oct 31 04:00:38.206246 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:00:38.206219    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7\": container with ID starting with 1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7 not found: ID does not exist" containerID="1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"
Oct 31 04:00:38.206394 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.206252    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7"} err="failed to get container status \"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7\": rpc error: code = NotFound desc = could not find container \"1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7\": container with ID starting with 1727ee29ae06a826679426b52331dad00f8a862d104d44d5f6815906749f45b7 not found: ID does not exist"
Oct 31 04:00:38.211060 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.211023    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 04:00:38.224450 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.224420    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-ztncr"]
Oct 31 04:00:38.243176 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.243138    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 04:00:38.253557 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:38.253517    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-bcc7a4-openshift-default-5d89c57df5-p2vjz"]
Oct 31 04:00:39.611272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:39.611231    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" path="/var/lib/kubelet/pods/d6c3302e-f961-43a9-b381-a9979bca3d1f/volumes"
Oct 31 04:00:39.611755 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:39.611740    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" path="/var/lib/kubelet/pods/dfa609e7-8ef4-41d3-a2da-3b7bf9dec882/volumes"
Oct 31 04:00:40.543190 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:40.543157    5098 scope.go:122] "RemoveContainer" containerID="5f7c88f93e86426ca1052ed683e8f0edaafaedf7d02de3f02f98c033394dea98"
Oct 31 04:00:40.553807 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:40.553779    5098 scope.go:122] "RemoveContainer" containerID="0727724ccaa942a5f9fb14f6b488bc46398dda413b932340ec06f859cf5b64a0"
Oct 31 04:00:40.564009 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:40.563984    5098 scope.go:122] "RemoveContainer" containerID="03153e251af1c1f809f39ef8a113f3066958959fdd26e4ccf0dafc5db519faa5"
Oct 31 04:00:40.574039 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:40.574007    5098 scope.go:122] "RemoveContainer" containerID="140be9d648c49f0ef777ab9e08101b81fc3987737d755f2cda2ba195b4a1cebf"
Oct 31 04:00:40.584164 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:00:40.584140    5098 scope.go:122] "RemoveContainer" containerID="80a0fbb6096eab6abee1c4c0878628d33a216c2a4121e025b63f9a983d74d48a"
Oct 31 04:01:00.972332 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.972293    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"]
Oct 31 04:01:00.973135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973103    5098 cpu_manager.go:414] "containerMap: removing container" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing"
Oct 31 04:01:00.973135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973131    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973152    5098 cpu_manager.go:414] "containerMap: removing container" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973162    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973178    5098 cpu_manager.go:414] "containerMap: removing container" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973185    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973197    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973205    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973220    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973227    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973242    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973247    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973257    5098 cpu_manager.go:414] "containerMap: removing container" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973261    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973270    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973274    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973286    5098 cpu_manager.go:414] "containerMap: removing container" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973291    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973299    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973303    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973316    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973323    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing"
Oct 31 04:01:00.973326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973335    5098 cpu_manager.go:414] "containerMap: removing container" podUID="29257438-7afd-4910-b02f-2a4581705f7d" containerName="revoke-keys"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973342    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29257438-7afd-4910-b02f-2a4581705f7d" containerName="revoke-keys"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973348    5098 cpu_manager.go:414] "containerMap: removing container" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973352    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973432    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973444    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab7e371-cb9c-46ca-be9c-faa8000ba33f" containerName="payload-pre-processing"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973450    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6c3302e-f961-43a9-b381-a9979bca3d1f" containerName="istio-proxy"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973458    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc210d02-f4a2-4f76-8a6f-12484b05d3ac" containerName="payload-processing"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973465    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="29257438-7afd-4910-b02f-2a4581705f7d" containerName="revoke-keys"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973472    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="69fad609-0f14-4d2c-8249-6e96a8ec3d30" containerName="main"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973482    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f9979b4-4472-4277-96cc-b5f430090bb2" containerName="payload-processing"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973488    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973493    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="510f30c0-9f70-4c49-b0c1-96fa9586aee8" containerName="main"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973499    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a4810ca-5db9-4232-a5fe-1abf2b78cc54" containerName="payload-pre-processing"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973505    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.974117 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.973510    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c046f55b-2491-441e-9099-f9df90217489" containerName="istio-proxy"
Oct 31 04:01:00.977342 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.977315    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:00.990233 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.990192    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"]
Oct 31 04:01:00.994476 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.994444    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"]
Oct 31 04:01:00.995202 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.995176    5098 cpu_manager.go:414] "containerMap: removing container" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.995344 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.995202    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.995344 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.995330    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb66b743-2410-4f2c-8f5f-1334d8759d08" containerName="cleanup"
Oct 31 04:01:00.995516 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.995351    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfa609e7-8ef4-41d3-a2da-3b7bf9dec882" containerName="istio-proxy"
Oct 31 04:01:00.999189 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:00.999169    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.015066 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.015037    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"]
Oct 31 04:01:01.057171 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:01:01.057131    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:01:01.066562 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066526    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.066756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066568    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.066756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066710    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2qfb\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-kube-api-access-q2qfb\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.066756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066747    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.066934 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066764    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.066934 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066917    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.067059 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.066956    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.067059 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.067038    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.067167 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.067125    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168028 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.167984    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168046    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168132    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2qfb\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-kube-api-access-q2qfb\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168155    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2cj\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-kube-api-access-bn2cj\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168180    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168197    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168409 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168380    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168443 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168423    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168476 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168439    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168521 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168500    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168572 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168542    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168659 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168598    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168655    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168727    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168852 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168814    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168912 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168863    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.168912 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168876    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.168912 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168893    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.169074 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.168939    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.169074 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.169050    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.169191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.169122    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.169191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.169152    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.169564 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.169529    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.171007 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.170984    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.171290 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.171258    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.176499 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.176474    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.176787 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.176765    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2qfb\" (UniqueName: \"kubernetes.io/projected/9a8b9637-0546-43b8-b5b4-c67bce1b0345-kube-api-access-q2qfb\") pod \"maas-default-gateway-openshift-default-769df7846c-6vdn7\" (UID: \"9a8b9637-0546-43b8-b5b4-c67bce1b0345\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.269639 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269523    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.269854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269684    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.269854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269732    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.269854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269775    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.269854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269806    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.269854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269844    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270163 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269875    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270163 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269913    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270163 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.269954    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2cj\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-kube-api-access-bn2cj\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270163 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.270092    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-credential-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270396 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.270265    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-data\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270396 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.270361    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270504 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.270401    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-certs\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.270764 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.270743    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-workload-socket\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.272487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.272455    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.272652 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.272635    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-envoy\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.278665 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.278627    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-istio-token\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.278919 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.278895    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2cj\" (UniqueName: \"kubernetes.io/projected/a50c6120-b67e-4ca2-a369-5ae4b01d5fdb-kube-api-access-bn2cj\") pod \"maas-default-gateway-openshift-default-769df7846c-tnsvh\" (UID: \"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.295438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.295409    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:01.316237 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.316201    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:01.590691 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.590654    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"]
Oct 31 04:01:01.591977 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:01:01.591942    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8b9637_0546_43b8_b5b4_c67bce1b0345.slice/crio-a86d2bcd58f09682f55c0b1ab476281922c023c439072a0173357155fa4e8c4f WatchSource:0}: Error finding container a86d2bcd58f09682f55c0b1ab476281922c023c439072a0173357155fa4e8c4f: Status 404 returned error can't find the container with id a86d2bcd58f09682f55c0b1ab476281922c023c439072a0173357155fa4e8c4f
Oct 31 04:01:01.594321 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.594287    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:01.594416 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.594362    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:01.594416 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.594396    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:01.621395 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.621362    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"]
Oct 31 04:01:01.628947 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:01:01.628907    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50c6120_b67e_4ca2_a369_5ae4b01d5fdb.slice/crio-5bca5dbf9882fde59d32bb82d6fa3f88fe897fd08a39ad06436f0d560acbec66 WatchSource:0}: Error finding container 5bca5dbf9882fde59d32bb82d6fa3f88fe897fd08a39ad06436f0d560acbec66: Status 404 returned error can't find the container with id 5bca5dbf9882fde59d32bb82d6fa3f88fe897fd08a39ad06436f0d560acbec66
Oct 31 04:01:01.631491 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.631457    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:01.631596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.631526    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:01.631596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:01.631552    5098 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"}
Oct 31 04:01:02.332002 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:02.331961    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" event={"ID":"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb","Type":"ContainerStarted","Data":"1627ed477c028b0531df3d58a7067a6b049acfe4753c3a1a920efc3a4f894bad"}
Oct 31 04:01:02.332500 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:02.332023    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" event={"ID":"a50c6120-b67e-4ca2-a369-5ae4b01d5fdb","Type":"ContainerStarted","Data":"5bca5dbf9882fde59d32bb82d6fa3f88fe897fd08a39ad06436f0d560acbec66"}
Oct 31 04:01:02.333934 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:02.333908    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" event={"ID":"9a8b9637-0546-43b8-b5b4-c67bce1b0345","Type":"ContainerStarted","Data":"e6c613138bdf024fe06e195541f272a7de57ca7b7998f328470f21e06e9363b4"}
Oct 31 04:01:02.334080 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:02.333955    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" event={"ID":"9a8b9637-0546-43b8-b5b4-c67bce1b0345","Type":"ContainerStarted","Data":"a86d2bcd58f09682f55c0b1ab476281922c023c439072a0173357155fa4e8c4f"}
Oct 31 04:01:02.367630 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:02.367480    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podStartSLOduration=2.367460262 podStartE2EDuration="2.367460262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:02.364926461 +0000 UTC m=+2736.890146064" watchObservedRunningTime="2026-10-31 04:01:02.367460262 +0000 UTC m=+2736.892679852"
Oct 31 04:01:03.298569 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.296852    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:03.301779 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.301743    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:03.301926 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.301814    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:03.319182 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.319140    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:03.320698 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.320667    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:03.320848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:03.320732    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:04.297655 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:04.296674    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:04.297655 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:04.296739    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:04.319430 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:04.319388    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:04.319648 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:04.319462    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:05.296233 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:05.296134    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:05.296703 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:05.296662    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:05.319593 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:05.319553    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:05.320164 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:05.319625    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:06.298675 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:06.298161    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:06.298675 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:06.298246    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:06.319786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:06.319745    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:06.320200 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:06.319803    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:07.297151 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:07.297102    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:07.298629 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:07.297431    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:07.319341 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:07.319304    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:07.319528 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:07.319361    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:08.296441 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:08.296367    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused" start-of-body=
Oct 31 04:01:08.297033 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:08.296466    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": dial tcp 10.132.1.46:15021: connect: connection refused"
Oct 31 04:01:08.319392 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:08.319334    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused" start-of-body=
Oct 31 04:01:08.319640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:08.319407    5098 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": dial tcp 10.132.1.47:15021: connect: connection refused"
Oct 31 04:01:10.097575 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.097532    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:10.098370 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.098345    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:10.099517 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.099493    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7"
Oct 31 04:01:10.106092 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.106059    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:10.106400 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.106378    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:10.107208 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.107186    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh"
Oct 31 04:01:10.123072 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:10.123013    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podStartSLOduration=10.12299379 podStartE2EDuration="10.12299379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:01:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:01:02.392923508 +0000 UTC m=+2736.918143126" watchObservedRunningTime="2026-10-31 04:01:10.12299379 +0000 UTC m=+2744.648213461"
Oct 31 04:01:31.057768 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:01:31.057725    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:01:34.023551 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.023511    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"]
Oct 31 04:01:34.026543 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.026515    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-535506ee-rfbbr"]
Oct 31 04:01:34.029232 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.029204    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"]
Oct 31 04:01:34.031466 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.031453    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"]
Oct 31 04:01:34.033826 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.033796    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"]
Oct 31 04:01:34.038571 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.038545    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29f66008-tm7xj"]
Oct 31 04:01:34.041843 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.041822    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27598604-fx9b5"]
Oct 31 04:01:34.044024 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:34.043999    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ebf6a69-5b9s7"]
Oct 31 04:01:35.611085 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:35.611051    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a7b6634-6adb-4a6e-95d1-747a689247d2" path="/var/lib/kubelet/pods/1a7b6634-6adb-4a6e-95d1-747a689247d2/volumes"
Oct 31 04:01:35.611521 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:35.611497    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63589c56-1abc-4ce6-945e-5198907c324d" path="/var/lib/kubelet/pods/63589c56-1abc-4ce6-945e-5198907c324d/volumes"
Oct 31 04:01:35.611948 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:35.611925    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c9e1dfc-f753-463e-ac40-060d52a1607b" path="/var/lib/kubelet/pods/6c9e1dfc-f753-463e-ac40-060d52a1607b/volumes"
Oct 31 04:01:35.612684 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:35.612667    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d" path="/var/lib/kubelet/pods/dae038ac-e8a2-4c0a-bf9a-4a65e2aaf53d/volumes"
Oct 31 04:01:39.022635 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.022571    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"]
Oct 31 04:01:39.026523 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.026491    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"]
Oct 31 04:01:39.028865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.028824    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0aad177f-pcsm4"]
Oct 31 04:01:39.030921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.030900    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da5422ce-ltmj9"]
Oct 31 04:01:39.610890 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.610851    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40d94e96-6a61-4a9e-9a6d-f5526a07a957" path="/var/lib/kubelet/pods/40d94e96-6a61-4a9e-9a6d-f5526a07a957/volumes"
Oct 31 04:01:39.611321 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:39.611306    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4fce454-2352-4351-89ee-219b65ba27aa" path="/var/lib/kubelet/pods/f4fce454-2352-4351-89ee-219b65ba27aa/volumes"
Oct 31 04:01:40.736155 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.736120    5098 scope.go:122] "RemoveContainer" containerID="0d21cf78c7a11713920b811a50a385fcd67f640f260735905d61d2b91bc7c6ed"
Oct 31 04:01:40.746678 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.746653    5098 scope.go:122] "RemoveContainer" containerID="daed7ed9edc111649be6866da33a5c7ced43e0449b8000cc1fd3199fb09f8294"
Oct 31 04:01:40.756784 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.756759    5098 scope.go:122] "RemoveContainer" containerID="a03e8ef459dbdb5f0077a29210c81b8fc182f1fa4f128f2cc8458383508e30df"
Oct 31 04:01:40.767219 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.767192    5098 scope.go:122] "RemoveContainer" containerID="ac90da13deaf7eab502759e2291bc6f258d3484e7d57a533e0cf058d2c769a27"
Oct 31 04:01:40.777268 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.777242    5098 scope.go:122] "RemoveContainer" containerID="efaeae4b94e7fa487c720eb5f906349e2c3d7c5573063dcb12b8386186010d51"
Oct 31 04:01:40.787186 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.787163    5098 scope.go:122] "RemoveContainer" containerID="ae37b6a1033649aa8c0219fb5070fa14a138548a08e293f999a7f1ca765b1ce3"
Oct 31 04:01:40.796586 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:40.796559    5098 scope.go:122] "RemoveContainer" containerID="62d3cea6e5efc13e5572809c96c4b03ad8b198a97c7b6479f94a2ff2a26287d2"
Oct 31 04:01:43.137039 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:01:43.137001    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:01.057957 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:02:01.057916    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:02:01.530947 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:01.530908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:05.948358 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:05.948312    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:20.326824 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:20.326788    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:28.017086 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:28.017047    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"]
Oct 31 04:02:28.019122 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:28.019097    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d0da114-x24g5"]
Oct 31 04:02:29.610324 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:29.610276    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef165c6c-c420-444c-a5dc-6c8b304227bb" path="/var/lib/kubelet/pods/ef165c6c-c420-444c-a5dc-6c8b304227bb/volumes"
Oct 31 04:02:31.057123 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:02:31.057082    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:02:32.931349 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:32.931312    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:40.151149 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.151066    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 04:02:40.151709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.151263    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" podUID="fe540f1a-81ec-4517-b816-fe5b54251eb1" containerName="discovery" containerID="cri-o://89d8e982c165a078acdeac0d885fb5a023f21d924a610618d7b150909eb4f43e" gracePeriod=30
Oct 31 04:02:40.419622 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.419588    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 04:02:40.476204 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476168    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476204 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476207    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476233    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476260    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476324    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476374    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476404    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\") pod \"fe540f1a-81ec-4517-b816-fe5b54251eb1\" (UID: \"fe540f1a-81ec-4517-b816-fe5b54251eb1\") "
Oct 31 04:02:40.476800 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476706    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:40.476866 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.476827    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-ca-configmap\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.479285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.479253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:02:40.479428 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.479313    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:40.479546 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.479521    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:40.479677 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.479650    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "kube-api-access-x7g8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:40.480697 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.480666    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:40.480804 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.480735    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts" pod "fe540f1a-81ec-4517-b816-fe5b54251eb1" (UID: "fe540f1a-81ec-4517-b816-fe5b54251eb1"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:40.577596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577556    5098 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-cacerts\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.577596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577586    5098 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-token\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.577596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577596    5098 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-csr-dns-cert\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.577596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577618    5098 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe540f1a-81ec-4517-b816-fe5b54251eb1-local-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.577906 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577627    5098 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fe540f1a-81ec-4517-b816-fe5b54251eb1-istio-kubeconfig\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.577906 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.577636    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7g8n\" (UniqueName: \"kubernetes.io/projected/fe540f1a-81ec-4517-b816-fe5b54251eb1-kube-api-access-x7g8n\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:40.855469 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.855433    5098 scope.go:122] "RemoveContainer" containerID="60d269a48a4db22514a784a36afac3f9d23770d9cc527112104a49b2b8dcc49a"
Oct 31 04:02:40.866709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.866681    5098 scope.go:122] "RemoveContainer" containerID="c9f02c142e8fb83f3d8f1e524cba84ef83a1a31a6220a3d61cc73be8ad8c8ace"
Oct 31 04:02:40.882408 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.882377    5098 scope.go:122] "RemoveContainer" containerID="63addc9cb88dfe52abce653c8b8bc2b37fc9f8ea856bca16905bb89fab3cde4a"
Oct 31 04:02:40.891947 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.891921    5098 scope.go:122] "RemoveContainer" containerID="8864a8caff0ed5c64dd33d72482f56234d0834608a70495715449cbab7ad2417"
Oct 31 04:02:40.903761 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.903735    5098 scope.go:122] "RemoveContainer" containerID="1e5bb85a2f5788936eb535efeed77122c2e4235c0210d748ef618fa6174a0999"
Oct 31 04:02:40.912785 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.912762    5098 scope.go:122] "RemoveContainer" containerID="c59f4fb5c2b8716bf1bed170d562203144b40c8ef55bf4f79da84204268a7f79"
Oct 31 04:02:40.921468 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.921442    5098 scope.go:122] "RemoveContainer" containerID="9284f84ee7c207713cfe7bd9dd48ef95b00bb25495834e13904f95a36a8c3e33"
Oct 31 04:02:40.930205 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.930181    5098 scope.go:122] "RemoveContainer" containerID="ad888ad844a4fd29e11e6c1099d0ef1ab09cb779229fdd77dd4a70471a98db90"
Oct 31 04:02:40.940069 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.940044    5098 scope.go:122] "RemoveContainer" containerID="30e7cbbf0d7a6a7916185f54d4e737cc4f7cc5a0ea812facff47346bdc66541b"
Oct 31 04:02:40.949508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.949487    5098 scope.go:122] "RemoveContainer" containerID="9723dee303dd842a18cb2b95f9105b4108f3e1cfdb0d21ebcd72cf465007cf8d"
Oct 31 04:02:40.958177 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.958158    5098 scope.go:122] "RemoveContainer" containerID="9eca71df12fdec2b95b3a11cb4a56d504fba90a8042fc3ea086d3a9657541ba8"
Oct 31 04:02:40.966838 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.966818    5098 scope.go:122] "RemoveContainer" containerID="7ac1f1fad601d7fe09ba3e781b89db9794de6b871abc979e5bb2dd08781eb513"
Oct 31 04:02:40.975333 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.975313    5098 scope.go:122] "RemoveContainer" containerID="9dbcaf19006ca0e65ca00c1823ab65a4dfa0bbb4d0c1ad4be1b6f0ecc198745d"
Oct 31 04:02:40.984018 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.983991    5098 scope.go:122] "RemoveContainer" containerID="274dc06924257190f7f7752a7b42a470244909a3b3ac30cadc9ce42159638126"
Oct 31 04:02:40.992851 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:40.992828    5098 scope.go:122] "RemoveContainer" containerID="88dcd74d8bff8570df443b09e8bb38781bf50f724b46798d07475dce8e37b1fa"
Oct 31 04:02:41.001706 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.001684    5098 scope.go:122] "RemoveContainer" containerID="89d8e982c165a078acdeac0d885fb5a023f21d924a610618d7b150909eb4f43e"
Oct 31 04:02:41.013289 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.013257    5098 scope.go:122] "RemoveContainer" containerID="564eda6acbec082e5c8d87bcc0fdc1472405a1db275933946b30db21a1b8154c"
Oct 31 04:02:41.153747 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.153661    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" event={"ID":"fe540f1a-81ec-4517-b816-fe5b54251eb1","Type":"ContainerDied","Data":"89d8e982c165a078acdeac0d885fb5a023f21d924a610618d7b150909eb4f43e"}
Oct 31 04:02:41.153747 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.153741    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8" event={"ID":"fe540f1a-81ec-4517-b816-fe5b54251eb1","Type":"ContainerDied","Data":"a07fe787312955e5ddc4d986f08e67369e7445de056dfc442c435007b0530a5f"}
Oct 31 04:02:41.154216 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.153789    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"
Oct 31 04:02:41.188994 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.188960    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 04:02:41.193943 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.193912    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qrrx8"]
Oct 31 04:02:41.611831 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:41.611787    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe540f1a-81ec-4517-b816-fe5b54251eb1" path="/var/lib/kubelet/pods/fe540f1a-81ec-4517-b816-fe5b54251eb1/volumes"
Oct 31 04:02:46.877119 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.877082    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"]
Oct 31 04:02:46.877574 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.877556    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fe540f1a-81ec-4517-b816-fe5b54251eb1" containerName="discovery"
Oct 31 04:02:46.877574 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.877568    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe540f1a-81ec-4517-b816-fe5b54251eb1" containerName="discovery"
Oct 31 04:02:46.877691 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.877660    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe540f1a-81ec-4517-b816-fe5b54251eb1" containerName="discovery"
Oct 31 04:02:46.883115 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.883091    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:46.888175 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.888143    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"]
Oct 31 04:02:46.888631 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.888586    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:02:46.888775 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.888695    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:02:46.889886 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.889866    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:02:46.890026 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.890011    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:02:46.941933 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.941893    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:46.941933 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.941934    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:46.942177 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:46.942140    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmqkw\" (UniqueName: \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.043021 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.042977    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmqkw\" (UniqueName: \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.043201 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.043049    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.043201 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.043073    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.043989 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.043964    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.045817 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.045790    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.053384 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.053355    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmqkw\" (UniqueName: \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\") pod \"maas-api-revoke-sub-046e1343-jph54\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.202699 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.202570    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:47.479405 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:47.479371    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"]
Oct 31 04:02:47.481281 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:02:47.481253    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b6ebf47_3da1_4e86_9ea1_9e383c783316.slice/crio-65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45 WatchSource:0}: Error finding container 65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45: Status 404 returned error can't find the container with id 65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45
Oct 31 04:02:48.196427 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:48.196383    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54" event={"ID":"3b6ebf47-3da1-4e86-9ea1-9e383c783316","Type":"ContainerStarted","Data":"1360b59f8726895db7313d632588b765895a5f59e3580008726d48fe594083ff"}
Oct 31 04:02:48.196874 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:48.196456    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54" event={"ID":"3b6ebf47-3da1-4e86-9ea1-9e383c783316","Type":"ContainerStarted","Data":"65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45"}
Oct 31 04:02:48.213566 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:48.213510    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54" podStartSLOduration=2.213495092 podStartE2EDuration="2.213495092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:48.211754035 +0000 UTC m=+2842.736973625" watchObservedRunningTime="2026-10-31 04:02:48.213495092 +0000 UTC m=+2842.738714680"
Oct 31 04:02:49.203811 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:49.203773    5098 generic.go:356] "Generic (PLEG): container finished" podID="3b6ebf47-3da1-4e86-9ea1-9e383c783316" containerID="1360b59f8726895db7313d632588b765895a5f59e3580008726d48fe594083ff" exitCode=0
Oct 31 04:02:49.204260 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:49.203834    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54" event={"ID":"3b6ebf47-3da1-4e86-9ea1-9e383c783316","Type":"ContainerDied","Data":"1360b59f8726895db7313d632588b765895a5f59e3580008726d48fe594083ff"}
Oct 31 04:02:50.360371 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.360339    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:50.477921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.477819    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\") pod \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") "
Oct 31 04:02:50.477921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.477879    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\") pod \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") "
Oct 31 04:02:50.477921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.477900    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\" (UniqueName: \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\") pod \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\" (UID: \"3b6ebf47-3da1-4e86-9ea1-9e383c783316\") "
Oct 31 04:02:50.478260 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.478233    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca" pod "3b6ebf47-3da1-4e86-9ea1-9e383c783316" (UID: "3b6ebf47-3da1-4e86-9ea1-9e383c783316"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:02:50.480140 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.480102    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca" pod "3b6ebf47-3da1-4e86-9ea1-9e383c783316" (UID: "3b6ebf47-3da1-4e86-9ea1-9e383c783316"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:02:50.480264 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.480236    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw" pod "3b6ebf47-3da1-4e86-9ea1-9e383c783316" (UID: "3b6ebf47-3da1-4e86-9ea1-9e383c783316"). InnerVolumeSpecName "kube-api-access-jmqkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:02:50.581797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.580421    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:50.581797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.580472    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b6ebf47-3da1-4e86-9ea1-9e383c783316-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:50.581797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:50.580490    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmqkw\" (UniqueName: \"kubernetes.io/projected/3b6ebf47-3da1-4e86-9ea1-9e383c783316-kube-api-access-jmqkw\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:02:51.218618 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:51.218576    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"
Oct 31 04:02:51.218618 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:51.218589    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54" event={"ID":"3b6ebf47-3da1-4e86-9ea1-9e383c783316","Type":"ContainerDied","Data":"65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45"}
Oct 31 04:02:51.218864 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:51.218656    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65305ef9551cab6189b8b5b068c8623a14082cd173a9d6b99f7af65a81c98b45"
Oct 31 04:02:57.072013 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.071968    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"]
Oct 31 04:02:57.072561 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.072443    5098 cpu_manager.go:414] "containerMap: removing container" podUID="3b6ebf47-3da1-4e86-9ea1-9e383c783316" containerName="revoke-keys"
Oct 31 04:02:57.072561 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.072461    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b6ebf47-3da1-4e86-9ea1-9e383c783316" containerName="revoke-keys"
Oct 31 04:02:57.072720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.072575    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b6ebf47-3da1-4e86-9ea1-9e383c783316" containerName="revoke-keys"
Oct 31 04:02:57.076070 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.076046    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.081371 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.081332    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:02:57.081773 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.081742    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:02:57.081916 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.081828    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:02:57.081975 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.081937    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:02:57.084070 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.084044    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"]
Oct 31 04:02:57.150051 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.150011    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.150229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.150091    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqrtm\" (UniqueName: \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.150229 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.150198    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.251217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.251184    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.251217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.251223    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqrtm\" (UniqueName: \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.251519 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.251264    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.251911 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.251882    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.254009 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.253985    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.259502 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.259475    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqrtm\" (UniqueName: \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\") pod \"maas-api-revoke-sub-d3fba056-l95pg\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.392501 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.392408    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:02:57.656374 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.656287    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:02:57.669245 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:57.669218    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"]
Oct 31 04:02:57.671280 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:02:57.671246    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab6c736_671e_4b02_999f_b53bf6597b5a.slice/crio-ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3 WatchSource:0}: Error finding container ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3: Status 404 returned error can't find the container with id ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3
Oct 31 04:02:58.264366 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:58.264262    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg" event={"ID":"8ab6c736-671e-4b02-999f-b53bf6597b5a","Type":"ContainerStarted","Data":"e3a3a57745f0a1992d1a5c1c6ff19d867fef3589fb2938b1143881752d5fb165"}
Oct 31 04:02:58.264366 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:58.264339    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg" event={"ID":"8ab6c736-671e-4b02-999f-b53bf6597b5a","Type":"ContainerStarted","Data":"ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3"}
Oct 31 04:02:58.282896 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:58.282835    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg" podStartSLOduration=1.282817837 podStartE2EDuration="1.282817837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:02:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:02:58.281886866 +0000 UTC m=+2852.807106459" watchObservedRunningTime="2026-10-31 04:02:58.282817837 +0000 UTC m=+2852.808037428"
Oct 31 04:02:59.321828 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:59.321782    5098 generic.go:356] "Generic (PLEG): container finished" podID="8ab6c736-671e-4b02-999f-b53bf6597b5a" containerID="e3a3a57745f0a1992d1a5c1c6ff19d867fef3589fb2938b1143881752d5fb165" exitCode=0
Oct 31 04:02:59.322673 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:02:59.322638    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg" event={"ID":"8ab6c736-671e-4b02-999f-b53bf6597b5a","Type":"ContainerDied","Data":"e3a3a57745f0a1992d1a5c1c6ff19d867fef3589fb2938b1143881752d5fb165"}
Oct 31 04:03:00.574827 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.574791    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:03:00.729991 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.729890    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\") pod \"8ab6c736-671e-4b02-999f-b53bf6597b5a\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") "
Oct 31 04:03:00.730153 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.730008    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\") pod \"8ab6c736-671e-4b02-999f-b53bf6597b5a\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") "
Oct 31 04:03:00.730153 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.730087    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\" (UniqueName: \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\") pod \"8ab6c736-671e-4b02-999f-b53bf6597b5a\" (UID: \"8ab6c736-671e-4b02-999f-b53bf6597b5a\") "
Oct 31 04:03:00.730453 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.730416    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca" pod "8ab6c736-671e-4b02-999f-b53bf6597b5a" (UID: "8ab6c736-671e-4b02-999f-b53bf6597b5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:00.732205 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.732178    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm" pod "8ab6c736-671e-4b02-999f-b53bf6597b5a" (UID: "8ab6c736-671e-4b02-999f-b53bf6597b5a"). InnerVolumeSpecName "kube-api-access-tqrtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:00.732300 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.732213    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca" pod "8ab6c736-671e-4b02-999f-b53bf6597b5a" (UID: "8ab6c736-671e-4b02-999f-b53bf6597b5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:00.831496 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.831446    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:00.831496 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.831488    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqrtm\" (UniqueName: \"kubernetes.io/projected/8ab6c736-671e-4b02-999f-b53bf6597b5a-kube-api-access-tqrtm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:00.831496 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:00.831504    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ab6c736-671e-4b02-999f-b53bf6597b5a-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:01.056953 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:03:01.056905    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:03:01.344576 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:01.344468    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg" event={"ID":"8ab6c736-671e-4b02-999f-b53bf6597b5a","Type":"ContainerDied","Data":"ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3"}
Oct 31 04:03:01.344918 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:01.344898    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffd616e208a4a11fe996021e60cc6e232c1db55c5962c6f04ea7184ca38b1ed3"
Oct 31 04:03:01.345118 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:01.345105    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"
Oct 31 04:03:07.957137 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:07.957099    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:13.023401 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:13.023365    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:17.892886 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.892851    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"]
Oct 31 04:03:17.893407 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.893390    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8ab6c736-671e-4b02-999f-b53bf6597b5a" containerName="revoke-keys"
Oct 31 04:03:17.893506 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.893408    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ab6c736-671e-4b02-999f-b53bf6597b5a" containerName="revoke-keys"
Oct 31 04:03:17.893567 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.893513    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ab6c736-671e-4b02-999f-b53bf6597b5a" containerName="revoke-keys"
Oct 31 04:03:17.897144 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.897119    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:17.902988 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.902961    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:03:17.903772 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.903020    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:03:17.903772 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.903087    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:03:17.903772 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.903115    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:03:17.905809 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.905772    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"]
Oct 31 04:03:17.983843 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.983807    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:17.984028 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.983858    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fn6m9\" (UniqueName: \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:17.984028 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:17.983941    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.085147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.085109    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.085369 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.085164    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fn6m9\" (UniqueName: \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.085369 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.085187    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.085835 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.085801    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.087895 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.087870    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.093769 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.093737    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fn6m9\" (UniqueName: \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\") pod \"maas-api-revoke-sub-ae594979-shxg6\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.215757 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.215663    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:18.494273 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:18.494245    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"]
Oct 31 04:03:18.496722 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:03:18.496688    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10da8251_284b_4078_8d0a_9df7ebae2edb.slice/crio-af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6 WatchSource:0}: Error finding container af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6: Status 404 returned error can't find the container with id af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6
Oct 31 04:03:19.474460 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:19.474424    5098 generic.go:356] "Generic (PLEG): container finished" podID="10da8251-284b-4078-8d0a-9df7ebae2edb" containerID="47bdcfa6ba2f9e5d013d9752174a2ee87c4de021b56cd14848498895f41d6e66" exitCode=0
Oct 31 04:03:19.474955 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:19.474521    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6" event={"ID":"10da8251-284b-4078-8d0a-9df7ebae2edb","Type":"ContainerDied","Data":"47bdcfa6ba2f9e5d013d9752174a2ee87c4de021b56cd14848498895f41d6e66"}
Oct 31 04:03:19.474955 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:19.474575    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6" event={"ID":"10da8251-284b-4078-8d0a-9df7ebae2edb","Type":"ContainerStarted","Data":"af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6"}
Oct 31 04:03:20.631338 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.631307    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:20.712011 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.711972    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\" (UniqueName: \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\") pod \"10da8251-284b-4078-8d0a-9df7ebae2edb\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") "
Oct 31 04:03:20.712234 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.712033    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\") pod \"10da8251-284b-4078-8d0a-9df7ebae2edb\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") "
Oct 31 04:03:20.712234 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.712090    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\") pod \"10da8251-284b-4078-8d0a-9df7ebae2edb\" (UID: \"10da8251-284b-4078-8d0a-9df7ebae2edb\") "
Oct 31 04:03:20.712495 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.712466    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca" pod "10da8251-284b-4078-8d0a-9df7ebae2edb" (UID: "10da8251-284b-4078-8d0a-9df7ebae2edb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:20.714284 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.714253    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca" pod "10da8251-284b-4078-8d0a-9df7ebae2edb" (UID: "10da8251-284b-4078-8d0a-9df7ebae2edb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:20.714394 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.714350    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9" pod "10da8251-284b-4078-8d0a-9df7ebae2edb" (UID: "10da8251-284b-4078-8d0a-9df7ebae2edb"). InnerVolumeSpecName "kube-api-access-fn6m9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:20.812919 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.812878    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:20.812919 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.812916    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fn6m9\" (UniqueName: \"kubernetes.io/projected/10da8251-284b-4078-8d0a-9df7ebae2edb-kube-api-access-fn6m9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:20.813145 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:20.812928    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10da8251-284b-4078-8d0a-9df7ebae2edb-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:21.498910 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:21.498870    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6" event={"ID":"10da8251-284b-4078-8d0a-9df7ebae2edb","Type":"ContainerDied","Data":"af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6"}
Oct 31 04:03:21.498910 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:21.498894    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"
Oct 31 04:03:21.499161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:21.498951    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af5a20ee500e13361431eb293dcc94501a5f3f1f53a7785355c0c0b62bf183a6"
Oct 31 04:03:28.666260 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:28.666216    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:31.063203 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:03:31.063154    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:03:32.622327 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:32.622284    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:35.945839 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.945798    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"]
Oct 31 04:03:35.946285 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.946270    5098 cpu_manager.go:414] "containerMap: removing container" podUID="10da8251-284b-4078-8d0a-9df7ebae2edb" containerName="revoke-keys"
Oct 31 04:03:35.946328 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.946286    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10da8251-284b-4078-8d0a-9df7ebae2edb" containerName="revoke-keys"
Oct 31 04:03:35.946372 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.946363    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="10da8251-284b-4078-8d0a-9df7ebae2edb" containerName="revoke-keys"
Oct 31 04:03:35.949751 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.949726    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:35.960014 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.959983    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:03:35.960179 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.960048    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:03:35.960261 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.960175    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:03:35.960324 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.960297    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:03:35.962150 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.962123    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"]
Oct 31 04:03:35.977297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.977260    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:35.977444 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.977416    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:35.977520 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:35.977451    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm68s\" (UniqueName: \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.078431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.078394    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.078431 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.078433    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm68s\" (UniqueName: \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.078735 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.078560    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.079166 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.079140    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.081185 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.081150    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.087035 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.087007    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm68s\" (UniqueName: \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\") pod \"maas-api-revoke-sub-a4aef7d7-7kvmk\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.265557 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.265474    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:36.541640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.541595    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"]
Oct 31 04:03:36.542529 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:03:36.542501    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abbf01e_4330_4aa0_81ff_15694d9b7fc0.slice/crio-057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e WatchSource:0}: Error finding container 057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e: Status 404 returned error can't find the container with id 057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e
Oct 31 04:03:36.629647 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:36.629585    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk" event={"ID":"4abbf01e-4330-4aa0-81ff-15694d9b7fc0","Type":"ContainerStarted","Data":"057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e"}
Oct 31 04:03:37.636801 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:37.636705    5098 generic.go:356] "Generic (PLEG): container finished" podID="4abbf01e-4330-4aa0-81ff-15694d9b7fc0" containerID="95149281258cf490e7ef3cadc6183a19e0f428e4ef4d63e1d764eb096a6643ed" exitCode=0
Oct 31 04:03:37.636801 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:37.636789    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk" event={"ID":"4abbf01e-4330-4aa0-81ff-15694d9b7fc0","Type":"ContainerDied","Data":"95149281258cf490e7ef3cadc6183a19e0f428e4ef4d63e1d764eb096a6643ed"}
Oct 31 04:03:38.797046 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.797014    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:38.914720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.914595    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\") pod \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") "
Oct 31 04:03:38.914720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.914695    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\") pod \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") "
Oct 31 04:03:38.914720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.914722    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\" (UniqueName: \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\") pod \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\" (UID: \"4abbf01e-4330-4aa0-81ff-15694d9b7fc0\") "
Oct 31 04:03:38.915128 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.915100    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca" pod "4abbf01e-4330-4aa0-81ff-15694d9b7fc0" (UID: "4abbf01e-4330-4aa0-81ff-15694d9b7fc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:03:38.916915 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.916885    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca" pod "4abbf01e-4330-4aa0-81ff-15694d9b7fc0" (UID: "4abbf01e-4330-4aa0-81ff-15694d9b7fc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:03:38.917026 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:38.916887    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s" pod "4abbf01e-4330-4aa0-81ff-15694d9b7fc0" (UID: "4abbf01e-4330-4aa0-81ff-15694d9b7fc0"). InnerVolumeSpecName "kube-api-access-jm68s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:03:39.016047 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.016009    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:39.016047 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.016039    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm68s\" (UniqueName: \"kubernetes.io/projected/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-kube-api-access-jm68s\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:39.016047 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.016050    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4abbf01e-4330-4aa0-81ff-15694d9b7fc0-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:03:39.650837 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.650808    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"
Oct 31 04:03:39.650837 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.650832    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk" event={"ID":"4abbf01e-4330-4aa0-81ff-15694d9b7fc0","Type":"ContainerDied","Data":"057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e"}
Oct 31 04:03:39.651057 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:39.650875    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="057554be4d0daa092c0d0eb9f171441ec29020c68685ebd9cd2999c430ef936e"
Oct 31 04:03:41.109789 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:41.109756    5098 scope.go:122] "RemoveContainer" containerID="4f5c67234304a0f7f4fce39b16969de1d11e57c14d2d471d07149ba1db509538"
Oct 31 04:03:41.119518 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:41.119498    5098 scope.go:122] "RemoveContainer" containerID="54c15c00c950e306bec9e40da313baed8dfe1321b84d0c3fe5516069a44dcc94"
Oct 31 04:03:41.128597 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:41.128572    5098 scope.go:122] "RemoveContainer" containerID="f659002372c2394771867cb6bc342dc94aa1e43d417fa7059457dddc7272dd51"
Oct 31 04:03:41.137373 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:41.137345    5098 scope.go:122] "RemoveContainer" containerID="b90cd3a631e56b3cbaf53584b0b87d7ecb1f4b8bda3355a99a4c87ed813d7a7f"
Oct 31 04:03:46.991652 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:46.991598    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:52.355518 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:52.355467    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:03:56.935894 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.935846    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"]
Oct 31 04:03:56.936483 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.936463    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4abbf01e-4330-4aa0-81ff-15694d9b7fc0" containerName="revoke-keys"
Oct 31 04:03:56.936542 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.936486    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abbf01e-4330-4aa0-81ff-15694d9b7fc0" containerName="revoke-keys"
Oct 31 04:03:56.936583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.936555    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abbf01e-4330-4aa0-81ff-15694d9b7fc0" containerName="revoke-keys"
Oct 31 04:03:56.939810 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.939789    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:56.943867 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.943837    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:03:56.944458 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.944413    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:03:56.945479 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.945442    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:03:56.945479 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.945466    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:03:56.949198 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:56.949170    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"]
Oct 31 04:03:57.000709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.000668    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.000932 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.000747    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zk2r\" (UniqueName: \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.000932 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.000838    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.102554 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.102510    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.102789 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.102600    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zk2r\" (UniqueName: \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.102789 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.102664    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.103379 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.103358    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.105049 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.105024    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.110878 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.110847    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zk2r\" (UniqueName: \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\") pod \"maas-api-revoke-sub-79805ca9-lffmt\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.255969 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.255883    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:03:57.530622 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.530577    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"]
Oct 31 04:03:57.531626 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:03:57.531580    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd372053_0e9e_43da_a696_77584c16abeb.slice/crio-60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e WatchSource:0}: Error finding container 60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e: Status 404 returned error can't find the container with id 60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e
Oct 31 04:03:57.763536 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:57.763494    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt" event={"ID":"fd372053-0e9e-43da-a696-77584c16abeb","Type":"ContainerStarted","Data":"60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e"}
Oct 31 04:03:58.774068 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:58.773972    5098 generic.go:356] "Generic (PLEG): container finished" podID="fd372053-0e9e-43da-a696-77584c16abeb" containerID="ad0161e45568a0e4e8a546857c974e49fa8b20ea87837f274e7c6983b726282a" exitCode=0
Oct 31 04:03:58.774458 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:58.774061    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt" event={"ID":"fd372053-0e9e-43da-a696-77584c16abeb","Type":"ContainerDied","Data":"ad0161e45568a0e4e8a546857c974e49fa8b20ea87837f274e7c6983b726282a"}
Oct 31 04:03:59.928896 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:03:59.928867    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:04:00.030749 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.030644    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\" (UniqueName: \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\") pod \"fd372053-0e9e-43da-a696-77584c16abeb\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") "
Oct 31 04:04:00.030749 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.030729    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\") pod \"fd372053-0e9e-43da-a696-77584c16abeb\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") "
Oct 31 04:04:00.030749 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.030750    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\") pod \"fd372053-0e9e-43da-a696-77584c16abeb\" (UID: \"fd372053-0e9e-43da-a696-77584c16abeb\") "
Oct 31 04:04:00.031236 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.031200    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca" pod "fd372053-0e9e-43da-a696-77584c16abeb" (UID: "fd372053-0e9e-43da-a696-77584c16abeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:00.033087 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.033051    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r" pod "fd372053-0e9e-43da-a696-77584c16abeb" (UID: "fd372053-0e9e-43da-a696-77584c16abeb"). InnerVolumeSpecName "kube-api-access-4zk2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:00.033263 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.033110    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca" pod "fd372053-0e9e-43da-a696-77584c16abeb" (UID: "fd372053-0e9e-43da-a696-77584c16abeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:04:00.132083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.132037    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zk2r\" (UniqueName: \"kubernetes.io/projected/fd372053-0e9e-43da-a696-77584c16abeb-kube-api-access-4zk2r\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:00.132083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.132076    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:00.132083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.132087    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd372053-0e9e-43da-a696-77584c16abeb-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:00.786083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.786049    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"
Oct 31 04:04:00.786278 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.786075    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt" event={"ID":"fd372053-0e9e-43da-a696-77584c16abeb","Type":"ContainerDied","Data":"60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e"}
Oct 31 04:04:00.786278 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:00.786128    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60861647ba9b3d88d77c2f4b5e54a094956f761dd7350ff1759a53a7d1cdca2e"
Oct 31 04:04:01.057262 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:04:01.057176    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:04:07.767324 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:07.767228    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:04:11.703576 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:11.703517    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:04:14.574811 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.574771    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"]
Oct 31 04:04:14.575465 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.575443    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fd372053-0e9e-43da-a696-77584c16abeb" containerName="revoke-keys"
Oct 31 04:04:14.575568 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.575468    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd372053-0e9e-43da-a696-77584c16abeb" containerName="revoke-keys"
Oct 31 04:04:14.575647 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.575592    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd372053-0e9e-43da-a696-77584c16abeb" containerName="revoke-keys"
Oct 31 04:04:14.579221 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.579199    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.584720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.584680    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:04:14.586106 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.586078    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:04:14.586272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.586186    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:04:14.587755 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.587736    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:04:14.590226 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.590201    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"]
Oct 31 04:04:14.696357 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.696318    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrgc\" (UniqueName: \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.696357 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.696369    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.696634 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.696494    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.797833 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.797800    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.798014 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.797922    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrgc\" (UniqueName: \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.798014 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.797947    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.798556 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.798534    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.800182 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.800160    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.806475 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.806451    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrgc\" (UniqueName: \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\") pod \"maas-api-revoke-sub-0b7e8edb-sjhvz\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:14.894932 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:14.894852    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:15.169651 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:15.169592    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"]
Oct 31 04:04:15.171334 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:04:15.171306    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717a4a1f_ed88_48c8_88d7_b8009dbdc137.slice/crio-14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341 WatchSource:0}: Error finding container 14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341: Status 404 returned error can't find the container with id 14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341
Oct 31 04:04:15.901885 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:15.901845    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz" event={"ID":"717a4a1f-ed88-48c8-88d7-b8009dbdc137","Type":"ContainerStarted","Data":"fd771e3ef51850769d3cf579115073f5156cc20f44cfcd9c8b1c20cc46595538"}
Oct 31 04:04:15.902317 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:15.901898    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz" event={"ID":"717a4a1f-ed88-48c8-88d7-b8009dbdc137","Type":"ContainerStarted","Data":"14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341"}
Oct 31 04:04:15.920096 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:15.920045    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz" podStartSLOduration=1.920031434 podStartE2EDuration="1.920031434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:04:15.917879503 +0000 UTC m=+2930.443099090" watchObservedRunningTime="2026-10-31 04:04:15.920031434 +0000 UTC m=+2930.445251072"
Oct 31 04:04:16.909081 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:16.908983    5098 generic.go:356] "Generic (PLEG): container finished" podID="717a4a1f-ed88-48c8-88d7-b8009dbdc137" containerID="fd771e3ef51850769d3cf579115073f5156cc20f44cfcd9c8b1c20cc46595538" exitCode=0
Oct 31 04:04:16.909480 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:16.909068    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz" event={"ID":"717a4a1f-ed88-48c8-88d7-b8009dbdc137","Type":"ContainerDied","Data":"fd771e3ef51850769d3cf579115073f5156cc20f44cfcd9c8b1c20cc46595538"}
Oct 31 04:04:18.078743 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.078716    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:18.234709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.234594    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\") pod \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") "
Oct 31 04:04:18.235005 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.234756    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\" (UniqueName: \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\") pod \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") "
Oct 31 04:04:18.235005 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.234784    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\") pod \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\" (UID: \"717a4a1f-ed88-48c8-88d7-b8009dbdc137\") "
Oct 31 04:04:18.235257 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.235226    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca" pod "717a4a1f-ed88-48c8-88d7-b8009dbdc137" (UID: "717a4a1f-ed88-48c8-88d7-b8009dbdc137"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:18.236841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.236811    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca" pod "717a4a1f-ed88-48c8-88d7-b8009dbdc137" (UID: "717a4a1f-ed88-48c8-88d7-b8009dbdc137"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:04:18.236841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.236832    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc" pod "717a4a1f-ed88-48c8-88d7-b8009dbdc137" (UID: "717a4a1f-ed88-48c8-88d7-b8009dbdc137"). InnerVolumeSpecName "kube-api-access-ghrgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:18.336191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.336149    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:18.336191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.336187    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrgc\" (UniqueName: \"kubernetes.io/projected/717a4a1f-ed88-48c8-88d7-b8009dbdc137-kube-api-access-ghrgc\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:18.336191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.336199    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/717a4a1f-ed88-48c8-88d7-b8009dbdc137-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:18.921840 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.921806    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"
Oct 31 04:04:18.922078 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.921848    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz" event={"ID":"717a4a1f-ed88-48c8-88d7-b8009dbdc137","Type":"ContainerDied","Data":"14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341"}
Oct 31 04:04:18.922078 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:18.921901    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14a5df2026faed10a33bce9d351fbb7c89817cbc91972cd4d3e03b1b9224a341"
Oct 31 04:04:25.239834 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.239798    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"]
Oct 31 04:04:25.240353 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.240334    5098 cpu_manager.go:414] "containerMap: removing container" podUID="717a4a1f-ed88-48c8-88d7-b8009dbdc137" containerName="revoke-keys"
Oct 31 04:04:25.240412 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.240354    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717a4a1f-ed88-48c8-88d7-b8009dbdc137" containerName="revoke-keys"
Oct 31 04:04:25.240469 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.240460    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="717a4a1f-ed88-48c8-88d7-b8009dbdc137" containerName="revoke-keys"
Oct 31 04:04:25.243822 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.243800    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.249461 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.249432    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:04:25.250297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.249554    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:04:25.250297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.249673    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:04:25.250297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.249764    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:04:25.253368 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.253344    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"]
Oct 31 04:04:25.403474 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.403427    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nlg\" (UniqueName: \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.403718 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.403504    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.403718 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.403694    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.439573 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.439538    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:04:25.504396 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.504302    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nlg\" (UniqueName: \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.504396 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.504392    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.504637 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.504454    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.505191 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.505167    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.506889 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.506864    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.512907 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.512882    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nlg\" (UniqueName: \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\") pod \"maas-api-revoke-sub-3d4156f1-g95jd\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.561175 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.561140    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:25.845596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.845563    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"]
Oct 31 04:04:25.848368 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:04:25.848333    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc178173d_cedc_4108_8e4a_5fc8580658da.slice/crio-9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc WatchSource:0}: Error finding container 9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc: Status 404 returned error can't find the container with id 9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc
Oct 31 04:04:25.964087 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:25.964047    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd" event={"ID":"c178173d-cedc-4108-8e4a-5fc8580658da","Type":"ContainerStarted","Data":"9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc"}
Oct 31 04:04:26.972616 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:26.972581    5098 generic.go:356] "Generic (PLEG): container finished" podID="c178173d-cedc-4108-8e4a-5fc8580658da" containerID="36cf8b97c2ddf6cc393fe153e9d6937a99d308deeee032330e24d79a4bf7c419" exitCode=0
Oct 31 04:04:26.973033 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:26.972647    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd" event={"ID":"c178173d-cedc-4108-8e4a-5fc8580658da","Type":"ContainerDied","Data":"36cf8b97c2ddf6cc393fe153e9d6937a99d308deeee032330e24d79a4bf7c419"}
Oct 31 04:04:28.121152 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.121123    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:28.142456 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.142422    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\") pod \"c178173d-cedc-4108-8e4a-5fc8580658da\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") "
Oct 31 04:04:28.142678 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.142516    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\") pod \"c178173d-cedc-4108-8e4a-5fc8580658da\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") "
Oct 31 04:04:28.142890 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.142865    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca" pod "c178173d-cedc-4108-8e4a-5fc8580658da" (UID: "c178173d-cedc-4108-8e4a-5fc8580658da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:28.144894 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.144867    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca" pod "c178173d-cedc-4108-8e4a-5fc8580658da" (UID: "c178173d-cedc-4108-8e4a-5fc8580658da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:04:28.243385 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.243276    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\" (UniqueName: \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\") pod \"c178173d-cedc-4108-8e4a-5fc8580658da\" (UID: \"c178173d-cedc-4108-8e4a-5fc8580658da\") "
Oct 31 04:04:28.243652 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.243601    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:28.243759 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.243657    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c178173d-cedc-4108-8e4a-5fc8580658da-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:28.245497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.245473    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg" pod "c178173d-cedc-4108-8e4a-5fc8580658da" (UID: "c178173d-cedc-4108-8e4a-5fc8580658da"). InnerVolumeSpecName "kube-api-access-b6nlg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:28.344171 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.344118    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nlg\" (UniqueName: \"kubernetes.io/projected/c178173d-cedc-4108-8e4a-5fc8580658da-kube-api-access-b6nlg\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:28.984550 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.984511    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd" event={"ID":"c178173d-cedc-4108-8e4a-5fc8580658da","Type":"ContainerDied","Data":"9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc"}
Oct 31 04:04:28.984550 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.984540    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"
Oct 31 04:04:28.984819 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:28.984567    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9881fdf35bb1ce51f81572b9e4d66768a5f6bcc92ba3fc4f11459049b40fddcc"
Oct 31 04:04:31.057986 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:04:31.057935    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:04:36.070416 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:36.070381    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:04:41.122794 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:41.122743    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:04:41.143830 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:41.143792    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:04:41.144103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:41.144077    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:04:41.170800 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:41.170756    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:04:41.171153 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:41.171103    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:04:46.345409 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.345365    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"]
Oct 31 04:04:46.352123 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.352082    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-g95jd"]
Oct 31 04:04:46.379850 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.379812    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"]
Oct 31 04:04:46.380479 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.380458    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c178173d-cedc-4108-8e4a-5fc8580658da" containerName="revoke-keys"
Oct 31 04:04:46.380561 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.380481    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c178173d-cedc-4108-8e4a-5fc8580658da" containerName="revoke-keys"
Oct 31 04:04:46.380637 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.380572    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c178173d-cedc-4108-8e4a-5fc8580658da" containerName="revoke-keys"
Oct 31 04:04:46.384297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.384272    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.392487 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.392415    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"]
Oct 31 04:04:46.392897 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.392872    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:04:46.393050 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.392917    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:04:46.393050 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.392971    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:04:46.393050 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.392988    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:04:46.483960 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.483923    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.484185 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.484056    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ptq7\" (UniqueName: \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.484185 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.484144    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.585049 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.585007    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ptq7\" (UniqueName: \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.585280 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.585058    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.585280 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.585160    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.585933 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.585901    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.587685 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.587657    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.594170 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.594136    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ptq7\" (UniqueName: \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\") pod \"maas-api-revoke-sub-3d4156f1-65tjf\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.704513 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.704428    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:46.982082 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:46.982045    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"]
Oct 31 04:04:46.984170 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:04:46.984136    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28b32a5_ac7a_456f_bc74_46a2da8f1782.slice/crio-80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e WatchSource:0}: Error finding container 80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e: Status 404 returned error can't find the container with id 80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e
Oct 31 04:04:47.127020 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:47.126981    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf" event={"ID":"f28b32a5-ac7a-456f-bc74-46a2da8f1782","Type":"ContainerStarted","Data":"80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e"}
Oct 31 04:04:47.610523 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:47.610483    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c178173d-cedc-4108-8e4a-5fc8580658da" path="/var/lib/kubelet/pods/c178173d-cedc-4108-8e4a-5fc8580658da/volumes"
Oct 31 04:04:48.137789 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:48.137699    5098 generic.go:356] "Generic (PLEG): container finished" podID="f28b32a5-ac7a-456f-bc74-46a2da8f1782" containerID="65dc84897fbdaefc772ee0df49d55475d92c67c29b0f70ae9305427336b1a0aa" exitCode=0
Oct 31 04:04:48.137939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:48.137784    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf" event={"ID":"f28b32a5-ac7a-456f-bc74-46a2da8f1782","Type":"ContainerDied","Data":"65dc84897fbdaefc772ee0df49d55475d92c67c29b0f70ae9305427336b1a0aa"}
Oct 31 04:04:49.290822 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.290791    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:49.309253 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.309217    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\" (UniqueName: \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\") pod \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") "
Oct 31 04:04:49.309460 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.309276    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\") pod \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") "
Oct 31 04:04:49.309460 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.309305    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\") pod \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\" (UID: \"f28b32a5-ac7a-456f-bc74-46a2da8f1782\") "
Oct 31 04:04:49.310151 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.310125    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca" pod "f28b32a5-ac7a-456f-bc74-46a2da8f1782" (UID: "f28b32a5-ac7a-456f-bc74-46a2da8f1782"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:04:49.312108 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.312082    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7" pod "f28b32a5-ac7a-456f-bc74-46a2da8f1782" (UID: "f28b32a5-ac7a-456f-bc74-46a2da8f1782"). InnerVolumeSpecName "kube-api-access-9ptq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:04:49.312239 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.312142    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca" pod "f28b32a5-ac7a-456f-bc74-46a2da8f1782" (UID: "f28b32a5-ac7a-456f-bc74-46a2da8f1782"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:04:49.411148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.411054    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ptq7\" (UniqueName: \"kubernetes.io/projected/f28b32a5-ac7a-456f-bc74-46a2da8f1782-kube-api-access-9ptq7\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:49.411148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.411095    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:49.411148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:49.411105    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28b32a5-ac7a-456f-bc74-46a2da8f1782-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:04:50.150672 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:50.150633    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"
Oct 31 04:04:50.150865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:50.150640    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf" event={"ID":"f28b32a5-ac7a-456f-bc74-46a2da8f1782","Type":"ContainerDied","Data":"80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e"}
Oct 31 04:04:50.150865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:50.150766    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80b65903e9c2c88c2194a32ab0d59e268a0cde507e522dbee12337839702486e"
Oct 31 04:04:54.025528 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:54.025486    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"]
Oct 31 04:04:54.033971 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:54.033667    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38ccbea0-676lb"]
Oct 31 04:04:55.613159 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:55.613109    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29b368a6-341b-40aa-b072-aee1a1427533" path="/var/lib/kubelet/pods/29b368a6-341b-40aa-b072-aee1a1427533/volumes"
Oct 31 04:04:57.664588 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:04:57.664551    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:01.059345 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:05:01.059307    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:05:01.518503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:01.518410    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:07.363950 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:07.363908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:14.025262 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.025221    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"]
Oct 31 04:05:14.027628 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.027589    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e39eef6-rz2x8"]
Oct 31 04:05:14.686148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.686112    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"]
Oct 31 04:05:14.686720 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.686698    5098 cpu_manager.go:414] "containerMap: removing container" podUID="f28b32a5-ac7a-456f-bc74-46a2da8f1782" containerName="revoke-keys"
Oct 31 04:05:14.686779 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.686720    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28b32a5-ac7a-456f-bc74-46a2da8f1782" containerName="revoke-keys"
Oct 31 04:05:14.686823 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.686802    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28b32a5-ac7a-456f-bc74-46a2da8f1782" containerName="revoke-keys"
Oct 31 04:05:14.690485 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.690466    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.695838 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.695815    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:05:14.696031 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.695847    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:05:14.696031 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.695866    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:05:14.696031 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.695882    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:05:14.698836 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.698814    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"]
Oct 31 04:05:14.782352 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.782313    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.782553 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.782445    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.782553 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.782520    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7nvp\" (UniqueName: \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.883664 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.883625    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.883861 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.883770    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7nvp\" (UniqueName: \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.883861 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.883850    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.884312 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.884291    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.886349 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.886324    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:14.892738 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:14.892711    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7nvp\" (UniqueName: \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\") pod \"maas-api-revoke-sub-e2fc9c5b-6kkph\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:15.005175 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:15.005074    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:15.283472 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:15.283438    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"]
Oct 31 04:05:15.284470 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:05:15.284443    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddba90cd0_3d85_47e7_beff_c6c191291dcc.slice/crio-6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740 WatchSource:0}: Error finding container 6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740: Status 404 returned error can't find the container with id 6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740
Oct 31 04:05:15.355444 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:15.355403    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph" event={"ID":"dba90cd0-3d85-47e7-beff-c6c191291dcc","Type":"ContainerStarted","Data":"6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740"}
Oct 31 04:05:15.610911 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:15.610869    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29257438-7afd-4910-b02f-2a4581705f7d" path="/var/lib/kubelet/pods/29257438-7afd-4910-b02f-2a4581705f7d/volumes"
Oct 31 04:05:16.367316 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:16.367224    5098 generic.go:356] "Generic (PLEG): container finished" podID="dba90cd0-3d85-47e7-beff-c6c191291dcc" containerID="450932934dbfdb8434167fe81dc3554d0a22cd95f92700ef2074e5eaf010f562" exitCode=0
Oct 31 04:05:16.367316 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:16.367270    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph" event={"ID":"dba90cd0-3d85-47e7-beff-c6c191291dcc","Type":"ContainerDied","Data":"450932934dbfdb8434167fe81dc3554d0a22cd95f92700ef2074e5eaf010f562"}
Oct 31 04:05:17.517525 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.517500    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:17.612700 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.612662    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\") pod \"dba90cd0-3d85-47e7-beff-c6c191291dcc\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") "
Oct 31 04:05:17.612896 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.612727    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\" (UniqueName: \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\") pod \"dba90cd0-3d85-47e7-beff-c6c191291dcc\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") "
Oct 31 04:05:17.612896 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.612803    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\") pod \"dba90cd0-3d85-47e7-beff-c6c191291dcc\" (UID: \"dba90cd0-3d85-47e7-beff-c6c191291dcc\") "
Oct 31 04:05:17.613069 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.613048    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca" pod "dba90cd0-3d85-47e7-beff-c6c191291dcc" (UID: "dba90cd0-3d85-47e7-beff-c6c191291dcc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:17.613150 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.613133    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:17.614887 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.614864    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca" pod "dba90cd0-3d85-47e7-beff-c6c191291dcc" (UID: "dba90cd0-3d85-47e7-beff-c6c191291dcc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:17.614961 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.614927    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp" pod "dba90cd0-3d85-47e7-beff-c6c191291dcc" (UID: "dba90cd0-3d85-47e7-beff-c6c191291dcc"). InnerVolumeSpecName "kube-api-access-s7nvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:17.713996 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.713901    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dba90cd0-3d85-47e7-beff-c6c191291dcc-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:17.713996 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:17.713935    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7nvp\" (UniqueName: \"kubernetes.io/projected/dba90cd0-3d85-47e7-beff-c6c191291dcc-kube-api-access-s7nvp\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:18.381253 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:18.381217    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"
Oct 31 04:05:18.381253 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:18.381239    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph" event={"ID":"dba90cd0-3d85-47e7-beff-c6c191291dcc","Type":"ContainerDied","Data":"6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740"}
Oct 31 04:05:18.381479 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:18.381290    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ce3ec80f89d799ad4ac19ff2d1440627c0f9d18851fbf9367b27b76cd5cc740"
Oct 31 04:05:24.884412 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.884372    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"]
Oct 31 04:05:24.884963 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.884845    5098 cpu_manager.go:414] "containerMap: removing container" podUID="dba90cd0-3d85-47e7-beff-c6c191291dcc" containerName="revoke-keys"
Oct 31 04:05:24.884963 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.884858    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dba90cd0-3d85-47e7-beff-c6c191291dcc" containerName="revoke-keys"
Oct 31 04:05:24.884963 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.884947    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="dba90cd0-3d85-47e7-beff-c6c191291dcc" containerName="revoke-keys"
Oct 31 04:05:24.888276 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.888244    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:24.893893 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.893721    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:05:24.893893 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.893796    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:05:24.893893 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.893839    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:05:24.893893 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.893863    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:05:24.896526 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.896234    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"]
Oct 31 04:05:24.985287 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.985246    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wswxl\" (UniqueName: \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:24.985489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.985315    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:24.985489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:24.985398    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.086765 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.086719    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wswxl\" (UniqueName: \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.086985 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.086790    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.086985 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.086843    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.087538 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.087514    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.089469 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.089442    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.094914 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.094885    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wswxl\" (UniqueName: \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\") pod \"maas-api-revoke-sub-5a29efb3-k7nwg\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.204939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.204832    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:25.484198 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.484161    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"]
Oct 31 04:05:25.486077 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:05:25.486051    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4587127b_c614_4ffc_87c9_a8315881896e.slice/crio-4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf WatchSource:0}: Error finding container 4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf: Status 404 returned error can't find the container with id 4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf
Oct 31 04:05:25.542484 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:25.542454    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:26.434390 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:26.434346    5098 generic.go:356] "Generic (PLEG): container finished" podID="4587127b-c614-4ffc-87c9-a8315881896e" containerID="d6a39d864cd52fbd47a749bc20f14d6e78d846790094a129e808eb0abf693384" exitCode=0
Oct 31 04:05:26.434915 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:26.434432    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg" event={"ID":"4587127b-c614-4ffc-87c9-a8315881896e","Type":"ContainerDied","Data":"d6a39d864cd52fbd47a749bc20f14d6e78d846790094a129e808eb0abf693384"}
Oct 31 04:05:26.434915 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:26.434502    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg" event={"ID":"4587127b-c614-4ffc-87c9-a8315881896e","Type":"ContainerStarted","Data":"4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf"}
Oct 31 04:05:27.857444 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.857064    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:27.937223 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.937173    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\") pod \"4587127b-c614-4ffc-87c9-a8315881896e\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") "
Oct 31 04:05:27.937445 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.937300    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\") pod \"4587127b-c614-4ffc-87c9-a8315881896e\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") "
Oct 31 04:05:27.937445 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.937390    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\" (UniqueName: \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\") pod \"4587127b-c614-4ffc-87c9-a8315881896e\" (UID: \"4587127b-c614-4ffc-87c9-a8315881896e\") "
Oct 31 04:05:27.938300 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.938271    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca" pod "4587127b-c614-4ffc-87c9-a8315881896e" (UID: "4587127b-c614-4ffc-87c9-a8315881896e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:27.940248 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.940219    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca" pod "4587127b-c614-4ffc-87c9-a8315881896e" (UID: "4587127b-c614-4ffc-87c9-a8315881896e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:27.940384 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:27.940339    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl" pod "4587127b-c614-4ffc-87c9-a8315881896e" (UID: "4587127b-c614-4ffc-87c9-a8315881896e"). InnerVolumeSpecName "kube-api-access-wswxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:28.038815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.038780    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:28.038815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.038811    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wswxl\" (UniqueName: \"kubernetes.io/projected/4587127b-c614-4ffc-87c9-a8315881896e-kube-api-access-wswxl\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:28.038815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.038823    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4587127b-c614-4ffc-87c9-a8315881896e-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:28.448506 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.448471    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg" event={"ID":"4587127b-c614-4ffc-87c9-a8315881896e","Type":"ContainerDied","Data":"4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf"}
Oct 31 04:05:28.448747 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.448528    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db36266f7bf9cd84d5c133a46701f12fa1c9fc23a9cb10db2a3a163594a22cf"
Oct 31 04:05:28.448747 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:28.448486    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"
Oct 31 04:05:31.057504 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:05:31.057464    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:05:33.016276 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:33.016244    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:05:33.023993 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:33.023956    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:05:35.931184 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:35.931098    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:40.420898 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.420857    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"]
Oct 31 04:05:40.427380 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.427352    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-65tjf"]
Oct 31 04:05:40.441400 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.441366    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"]
Oct 31 04:05:40.441874 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.441860    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4587127b-c614-4ffc-87c9-a8315881896e" containerName="revoke-keys"
Oct 31 04:05:40.441949 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.441876    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4587127b-c614-4ffc-87c9-a8315881896e" containerName="revoke-keys"
Oct 31 04:05:40.442013 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.441966    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4587127b-c614-4ffc-87c9-a8315881896e" containerName="revoke-keys"
Oct 31 04:05:40.445255 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.445231    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.450551 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.450522    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:05:40.450551 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.450535    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:05:40.450809 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.450567    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:05:40.450948 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.450926    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:05:40.453426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.453389    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"]
Oct 31 04:05:40.569492 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.569451    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.569819 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.569514    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.569819 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.569689    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8x88\" (UniqueName: \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.670351 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.670308    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.670554 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.670378    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8x88\" (UniqueName: \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.670554 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.670485    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.671217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.671171    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.672894 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.672865    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.678432 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.678400    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8x88\" (UniqueName: \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\") pod \"maas-api-revoke-sub-3d4156f1-s96z2\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:40.764368 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:40.764328    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:41.040795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.040761    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"]
Oct 31 04:05:41.042797 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:05:41.042772    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540754d4_32cf_4283_a75c_9585442011b8.slice/crio-f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c WatchSource:0}: Error finding container f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c: Status 404 returned error can't find the container with id f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c
Oct 31 04:05:41.246979 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.246942    5098 scope.go:122] "RemoveContainer" containerID="61c3e8be68eeb495f44e56e1eb93e7598b732cb2885fcbcfeec2b02fc757b327"
Oct 31 04:05:41.534149 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.534104    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2" event={"ID":"540754d4-32cf-4283-a75c-9585442011b8","Type":"ContainerStarted","Data":"39c452de1ca2a9c5762f16c8bf57855edf67b0011c5acc03c8920fb5e77cb9df"}
Oct 31 04:05:41.534640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.534202    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2" event={"ID":"540754d4-32cf-4283-a75c-9585442011b8","Type":"ContainerStarted","Data":"f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c"}
Oct 31 04:05:41.554525 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.554459    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2" podStartSLOduration=1.554444696 podStartE2EDuration="1.554444696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:05:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:05:41.552152228 +0000 UTC m=+3016.077371818" watchObservedRunningTime="2026-10-31 04:05:41.554444696 +0000 UTC m=+3016.079664284"
Oct 31 04:05:41.610589 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:41.610503    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28b32a5-ac7a-456f-bc74-46a2da8f1782" path="/var/lib/kubelet/pods/f28b32a5-ac7a-456f-bc74-46a2da8f1782/volumes"
Oct 31 04:05:42.541785 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:42.541749    5098 generic.go:356] "Generic (PLEG): container finished" podID="540754d4-32cf-4283-a75c-9585442011b8" containerID="39c452de1ca2a9c5762f16c8bf57855edf67b0011c5acc03c8920fb5e77cb9df" exitCode=0
Oct 31 04:05:42.542188 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:42.541835    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2" event={"ID":"540754d4-32cf-4283-a75c-9585442011b8","Type":"ContainerDied","Data":"39c452de1ca2a9c5762f16c8bf57855edf67b0011c5acc03c8920fb5e77cb9df"}
Oct 31 04:05:43.699054 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.699027    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:43.803974 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.803877    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\") pod \"540754d4-32cf-4283-a75c-9585442011b8\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") "
Oct 31 04:05:43.803974 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.803923    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\") pod \"540754d4-32cf-4283-a75c-9585442011b8\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") "
Oct 31 04:05:43.804186 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.803982    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\" (UniqueName: \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\") pod \"540754d4-32cf-4283-a75c-9585442011b8\" (UID: \"540754d4-32cf-4283-a75c-9585442011b8\") "
Oct 31 04:05:43.804360 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.804335    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca" pod "540754d4-32cf-4283-a75c-9585442011b8" (UID: "540754d4-32cf-4283-a75c-9585442011b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:05:43.806156 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.806132    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca" pod "540754d4-32cf-4283-a75c-9585442011b8" (UID: "540754d4-32cf-4283-a75c-9585442011b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:05:43.806282 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.806266    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88" pod "540754d4-32cf-4283-a75c-9585442011b8" (UID: "540754d4-32cf-4283-a75c-9585442011b8"). InnerVolumeSpecName "kube-api-access-b8x88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:05:43.905217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.905170    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/540754d4-32cf-4283-a75c-9585442011b8-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:43.905217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.905208    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/540754d4-32cf-4283-a75c-9585442011b8-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:43.905217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:43.905220    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8x88\" (UniqueName: \"kubernetes.io/projected/540754d4-32cf-4283-a75c-9585442011b8-kube-api-access-b8x88\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:05:44.554347 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:44.554314    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"
Oct 31 04:05:44.554546 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:44.554351    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2" event={"ID":"540754d4-32cf-4283-a75c-9585442011b8","Type":"ContainerDied","Data":"f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c"}
Oct 31 04:05:44.554546 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:44.554400    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1fd4a40b6564e944a96c6712f8921ec2355d428b3d021495f2499845f7b273c"
Oct 31 04:05:51.321959 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:51.321917    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:05:55.565590 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:05:55.565548    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:06:01.057182 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:06:01.057138    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:06:03.506503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.504867    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:06:03.506503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.505318    5098 cpu_manager.go:414] "containerMap: removing container" podUID="540754d4-32cf-4283-a75c-9585442011b8" containerName="revoke-keys"
Oct 31 04:06:03.506503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.505332    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="540754d4-32cf-4283-a75c-9585442011b8" containerName="revoke-keys"
Oct 31 04:06:03.506503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.505436    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="540754d4-32cf-4283-a75c-9585442011b8" containerName="revoke-keys"
Oct 31 04:06:03.509400 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.509372    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.517467 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.517434    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-799962-serving-cert\""
Oct 31 04:06:03.520148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.520121    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:06:03.520967 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.520946    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\""
Oct 31 04:06:03.521224 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.521204    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:06:03.521334 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.521275    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:06:03.523638 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.523588    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:06:03.537131 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.537086    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6st\" (UniqueName: \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.537297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.537212    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.537361 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.537335    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.537426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.537412    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.638897 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.638856    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.639105 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.638918    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.639105 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.638968    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.639105 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.639035    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6st\" (UniqueName: \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.639715 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.639682    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.641570 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.641542    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.641716 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.641590    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.651207 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.651180    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6st\" (UniqueName: \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\") pod \"maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.721961 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.721928    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:06:03.725888 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.725868    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.733976 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.733948    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:06:03.803662 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.803362    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0bd916eb-dockercfg-kkpcl\""
Oct 31 04:06:03.803878 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.803688    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-799962\""
Oct 31 04:06:03.841489 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.841450    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.841706 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.841500    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tq8\" (UniqueName: \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.843265 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.843242    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:03.943409 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.943355    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.944898 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.943436    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tq8\" (UniqueName: \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.944898 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.944789    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:03.954845 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:03.954789    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tq8\" (UniqueName: \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\") pod \"payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:04.044830 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.044761    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:04.138733 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.133680    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:06:04.142482 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:06:04.142440    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa572736_0fb1_4f6b_9800_475af68ca329.slice/crio-0ddef321e828de0305e406f074c9aa4f936c207dcb407df60c6596d84db27c82 WatchSource:0}: Error finding container 0ddef321e828de0305e406f074c9aa4f936c207dcb407df60c6596d84db27c82: Status 404 returned error can't find the container with id 0ddef321e828de0305e406f074c9aa4f936c207dcb407df60c6596d84db27c82
Oct 31 04:06:04.144583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.144540    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:06:04.346698 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.346666    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:06:04.348007 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:06:04.347976    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1220ff4_92fd_41aa_a7f8_65da416dd070.slice/crio-d2c1638e5b3a8fc923301518febfb6f018e2b2936e0d97fa6ffc9d1f7a7c362e WatchSource:0}: Error finding container d2c1638e5b3a8fc923301518febfb6f018e2b2936e0d97fa6ffc9d1f7a7c362e: Status 404 returned error can't find the container with id d2c1638e5b3a8fc923301518febfb6f018e2b2936e0d97fa6ffc9d1f7a7c362e
Oct 31 04:06:04.698976 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.698936    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" event={"ID":"fa572736-0fb1-4f6b-9800-475af68ca329","Type":"ContainerStarted","Data":"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"}
Oct 31 04:06:04.699426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.699007    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" event={"ID":"fa572736-0fb1-4f6b-9800-475af68ca329","Type":"ContainerStarted","Data":"0ddef321e828de0305e406f074c9aa4f936c207dcb407df60c6596d84db27c82"}
Oct 31 04:06:04.699426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.699150    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:04.700724 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.700700    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" event={"ID":"b1220ff4-92fd-41aa-a7f8-65da416dd070","Type":"ContainerStarted","Data":"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"}
Oct 31 04:06:04.700857 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.700738    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" event={"ID":"b1220ff4-92fd-41aa-a7f8-65da416dd070","Type":"ContainerStarted","Data":"d2c1638e5b3a8fc923301518febfb6f018e2b2936e0d97fa6ffc9d1f7a7c362e"}
Oct 31 04:06:04.700857 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.700770    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:04.719953 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.719896    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" podStartSLOduration=1.289532723 podStartE2EDuration="1.719880148s" totalImagesPullingTime="430.347425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:04.71711008 +0000 UTC m=+3039.242329717" watchObservedRunningTime="2026-10-31 04:06:04.719880148 +0000 UTC m=+3039.245099738"
Oct 31 04:06:04.734275 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:04.734172    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" podStartSLOduration=1.734153827 podStartE2EDuration="1.734153827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:04.733574357 +0000 UTC m=+3039.258793969" watchObservedRunningTime="2026-10-31 04:06:04.734153827 +0000 UTC m=+3039.259373419"
Oct 31 04:06:05.867844 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:05.867797    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:06:10.715875 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:10.715842    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:06:14.528673 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.528623    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:06:14.533490 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.533463    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.543448 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.543412    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-799962\"/\"openshift-service-ca.crt\""
Oct 31 04:06:14.543666 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.543568    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-799962\"/\"kube-root-ca.crt\""
Oct 31 04:06:14.546680 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.546652    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:06:14.563367 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.563340    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-799962\"/\"default-dockercfg-zrtz5\""
Oct 31 04:06:14.563539 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.563457    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-799962\"/\"simulator-single-main-799962-kserve-self-signed-certs\""
Oct 31 04:06:14.656922 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.656889    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.657135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.657056    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.657135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.657099    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.657135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.657128    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.657316 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.657296    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pbv2\" (UniqueName: \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.657392 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.657373    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758381    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758660 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758438    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758660 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758464    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758660 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758500    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pbv2\" (UniqueName: \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758660 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758566    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758977 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758768    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.758977 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.758879    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.759104 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.759089    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.759410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.759379    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.761570 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.761543    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.761999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.761981    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.767353 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.767326    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pbv2\" (UniqueName: \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\") pod \"simulator-single-main-799962-kserve-74f9d89b84-msjtd\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") " pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:14.849115 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:14.849033    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:15.129276 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.129241    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:06:15.130029 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:06:15.130001    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podec32b545_8a86_4538_a109_130a12a70cee.slice/crio-a001b051d072e54d27ac7b814a513ab9d9408b7ff75e64826b2197b785a3cb5e WatchSource:0}: Error finding container a001b051d072e54d27ac7b814a513ab9d9408b7ff75e64826b2197b785a3cb5e: Status 404 returned error can't find the container with id a001b051d072e54d27ac7b814a513ab9d9408b7ff75e64826b2197b785a3cb5e
Oct 31 04:06:15.713424 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.713384    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:06:15.774002 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.773967    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" event={"ID":"ec32b545-8a86-4538-a109-130a12a70cee","Type":"ContainerStarted","Data":"82eb13c6535848ee73817819f1c3f72eb5e5eb54051e422cf55a44313b2ef351"}
Oct 31 04:06:15.774175 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.774019    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" event={"ID":"ec32b545-8a86-4538-a109-130a12a70cee","Type":"ContainerStarted","Data":"a001b051d072e54d27ac7b814a513ab9d9408b7ff75e64826b2197b785a3cb5e"}
Oct 31 04:06:15.793675 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.793620    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" podStartSLOduration=1.793592767 podStartE2EDuration="1.793592767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:06:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:06:15.79118629 +0000 UTC m=+3050.316405880" watchObservedRunningTime="2026-10-31 04:06:15.793592767 +0000 UTC m=+3050.318812389"
Oct 31 04:06:15.849874 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:15.849832    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:16.077192 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:16.077154    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:06:24.849244 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:24.849209    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:24.862772 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:24.862745    5098 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:25.849268 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:25.849235    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:06:31.057761 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:06:31.057720    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:06:32.670998 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:32.670957    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:06:41.273334 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:41.273301    5098 scope.go:122] "RemoveContainer" containerID="1fba6328b12b853a3c499b2a5701cdfed007fc11121165b4e1e09cbb735f1637"
Oct 31 04:06:41.283859 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:41.283835    5098 scope.go:122] "RemoveContainer" containerID="1560671dca9a86100188de39923e351e96e9a0cc9a28fbd4c58fac392bed9f5b"
Oct 31 04:06:55.729482 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:55.729444    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:06:59.954986 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:06:59.954945    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:07:01.073571 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:07:01.073524    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:07:05.249219 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:05.249117    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:07:06.298418 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.298377    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:07:06.298978 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.298670    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" podUID="fa572736-0fb1-4f6b-9800-475af68ca329" containerName="maas-api" containerID="cri-o://edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3" gracePeriod=30
Oct 31 04:07:06.763715 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.763675    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:07:06.895508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.895406    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\") pod \"fa572736-0fb1-4f6b-9800-475af68ca329\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") "
Oct 31 04:07:06.895508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.895460    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\" (UniqueName: \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\") pod \"fa572736-0fb1-4f6b-9800-475af68ca329\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") "
Oct 31 04:07:06.895508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.895494    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\") pod \"fa572736-0fb1-4f6b-9800-475af68ca329\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") "
Oct 31 04:07:06.895877 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.895571    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\") pod \"fa572736-0fb1-4f6b-9800-475af68ca329\" (UID: \"fa572736-0fb1-4f6b-9800-475af68ca329\") "
Oct 31 04:07:06.896042 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.896014    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca" pod "fa572736-0fb1-4f6b-9800-475af68ca329" (UID: "fa572736-0fb1-4f6b-9800-475af68ca329"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:07:06.897803 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.897767    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs" pod "fa572736-0fb1-4f6b-9800-475af68ca329" (UID: "fa572736-0fb1-4f6b-9800-475af68ca329"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:07:06.898189 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.898156    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls" pod "fa572736-0fb1-4f6b-9800-475af68ca329" (UID: "fa572736-0fb1-4f6b-9800-475af68ca329"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:07:06.898189 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.898171    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st" pod "fa572736-0fb1-4f6b-9800-475af68ca329" (UID: "fa572736-0fb1-4f6b-9800-475af68ca329"). InnerVolumeSpecName "kube-api-access-dk6st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:06.997003 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.996961    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa572736-0fb1-4f6b-9800-475af68ca329-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:06.997003 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.996995    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:06.997003 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.997005    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk6st\" (UniqueName: \"kubernetes.io/projected/fa572736-0fb1-4f6b-9800-475af68ca329-kube-api-access-dk6st\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:06.997269 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:06.997015    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa572736-0fb1-4f6b-9800-475af68ca329-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:07.109094 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.109058    5098 generic.go:356] "Generic (PLEG): container finished" podID="fa572736-0fb1-4f6b-9800-475af68ca329" containerID="edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3" exitCode=0
Oct 31 04:07:07.109272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.109108    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" event={"ID":"fa572736-0fb1-4f6b-9800-475af68ca329","Type":"ContainerDied","Data":"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"}
Oct 31 04:07:07.109272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.109111    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"
Oct 31 04:07:07.109272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.109148    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm" event={"ID":"fa572736-0fb1-4f6b-9800-475af68ca329","Type":"ContainerDied","Data":"0ddef321e828de0305e406f074c9aa4f936c207dcb407df60c6596d84db27c82"}
Oct 31 04:07:07.109272 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.109172    5098 scope.go:122] "RemoveContainer" containerID="edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"
Oct 31 04:07:07.120454 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.120432    5098 scope.go:122] "RemoveContainer" containerID="edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"
Oct 31 04:07:07.120750 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:07:07.120727    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3\": container with ID starting with edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3 not found: ID does not exist" containerID="edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"
Oct 31 04:07:07.120809 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.120761    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3"} err="failed to get container status \"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3\": rpc error: code = NotFound desc = could not find container \"edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3\": container with ID starting with edbec64acccffa8718177fd1f18161850755193c13c5842e124b6c458b6d23f3 not found: ID does not exist"
Oct 31 04:07:07.134207 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.134167    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:07:07.138092 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.138062    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-799962-77b579d4bd-m8jwm"]
Oct 31 04:07:07.610864 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:07.610826    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa572736-0fb1-4f6b-9800-475af68ca329" path="/var/lib/kubelet/pods/fa572736-0fb1-4f6b-9800-475af68ca329/volumes"
Oct 31 04:07:08.034178 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:08.034136    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:07:10.249848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:10.249797    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing" containerID="cri-o://b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61" gracePeriod=25
Oct 31 04:07:14.370865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.370820    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"]
Oct 31 04:07:14.375226 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.375191    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-s96z2"]
Oct 31 04:07:14.392339 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.392308    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"]
Oct 31 04:07:14.392860 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.392836    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fa572736-0fb1-4f6b-9800-475af68ca329" containerName="maas-api"
Oct 31 04:07:14.392860 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.392856    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa572736-0fb1-4f6b-9800-475af68ca329" containerName="maas-api"
Oct 31 04:07:14.393111 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.393019    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa572736-0fb1-4f6b-9800-475af68ca329" containerName="maas-api"
Oct 31 04:07:14.397900 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.397877    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.401779 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.401749    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:07:14.403144 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.403118    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:07:14.403144 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.403128    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:07:14.403326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.403128    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:07:14.403326 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.403184    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"]
Oct 31 04:07:14.578569 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.578533    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.578790 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.578582    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.578790 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.578702    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjf8r\" (UniqueName: \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.680093 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.679999    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjf8r\" (UniqueName: \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.680266 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.680112    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.680266 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.680145    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.680912 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.680892    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.682588 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.682571    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.702897 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.702868    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjf8r\" (UniqueName: \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\") pod \"maas-api-revoke-sub-3d4156f1-64p2h\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.715736 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.715709    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:14.976317 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.976195    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" podUID="ec32b545-8a86-4538-a109-130a12a70cee" containerName="main" containerID="cri-o://82eb13c6535848ee73817819f1c3f72eb5e5eb54051e422cf55a44313b2ef351" gracePeriod=45
Oct 31 04:07:14.997593 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:14.997563    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"]
Oct 31 04:07:15.049848 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:07:15.049811    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b45bf1_c805_48ab_8578_9158e2736811.slice/crio-158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0 WatchSource:0}: Error finding container 158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0: Status 404 returned error can't find the container with id 158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0
Oct 31 04:07:15.171454 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.171415    5098 generic.go:356] "Generic (PLEG): container finished" podID="ec32b545-8a86-4538-a109-130a12a70cee" containerID="82eb13c6535848ee73817819f1c3f72eb5e5eb54051e422cf55a44313b2ef351" exitCode=0
Oct 31 04:07:15.171698 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.171491    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" event={"ID":"ec32b545-8a86-4538-a109-130a12a70cee","Type":"ContainerDied","Data":"82eb13c6535848ee73817819f1c3f72eb5e5eb54051e422cf55a44313b2ef351"}
Oct 31 04:07:15.173438 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.173398    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h" event={"ID":"91b45bf1-c805-48ab-8578-9158e2736811","Type":"ContainerStarted","Data":"158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0"}
Oct 31 04:07:15.270501 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.270467    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:07:15.389462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389414    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.389462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389467    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\" (UniqueName: \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389515    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389545    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389567    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389649    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\") pod \"ec32b545-8a86-4538-a109-130a12a70cee\" (UID: \"ec32b545-8a86-4538-a109-130a12a70cee\") "
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389761    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389891    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:07:15.390043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.389905    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:07:15.390550 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.390067    5098 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-home\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.390550 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.390081    5098 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-model-cache\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.390550 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.390090    5098 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-tmp-dir\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.392001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.391970    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 04:07:15.392128 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.391997    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:07:15.392128 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.392020    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2" pod "ec32b545-8a86-4538-a109-130a12a70cee" (UID: "ec32b545-8a86-4538-a109-130a12a70cee"). InnerVolumeSpecName "kube-api-access-5pbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:15.490648 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.490601    5098 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec32b545-8a86-4538-a109-130a12a70cee-tls-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.490648 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.490646    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pbv2\" (UniqueName: \"kubernetes.io/projected/ec32b545-8a86-4538-a109-130a12a70cee-kube-api-access-5pbv2\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.490648 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.490657    5098 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ec32b545-8a86-4538-a109-130a12a70cee-dshm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:15.610450 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.610404    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="540754d4-32cf-4283-a75c-9585442011b8" path="/var/lib/kubelet/pods/540754d4-32cf-4283-a75c-9585442011b8/volumes"
Oct 31 04:07:15.711376 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:15.711337    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:07:16.182126 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.182006    5098 generic.go:356] "Generic (PLEG): container finished" podID="91b45bf1-c805-48ab-8578-9158e2736811" containerID="cb081fbaf3e3d20d3a7d682fcdbcb66b2e0baac1e56d56b9c11ee03a3bc78502" exitCode=0
Oct 31 04:07:16.182126 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.182091    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h" event={"ID":"91b45bf1-c805-48ab-8578-9158e2736811","Type":"ContainerDied","Data":"cb081fbaf3e3d20d3a7d682fcdbcb66b2e0baac1e56d56b9c11ee03a3bc78502"}
Oct 31 04:07:16.183656 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.183631    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd" event={"ID":"ec32b545-8a86-4538-a109-130a12a70cee","Type":"ContainerDied","Data":"a001b051d072e54d27ac7b814a513ab9d9408b7ff75e64826b2197b785a3cb5e"}
Oct 31 04:07:16.183656 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.183654    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"
Oct 31 04:07:16.183831 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.183676    5098 scope.go:122] "RemoveContainer" containerID="82eb13c6535848ee73817819f1c3f72eb5e5eb54051e422cf55a44313b2ef351"
Oct 31 04:07:16.241590 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.241548    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:07:16.254144 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:16.254112    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-799962/simulator-single-main-799962-kserve-74f9d89b84-msjtd"]
Oct 31 04:07:17.334112 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.334084    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:17.411917 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.411877    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\") pod \"91b45bf1-c805-48ab-8578-9158e2736811\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") "
Oct 31 04:07:17.412131 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.411945    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\" (UniqueName: \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\") pod \"91b45bf1-c805-48ab-8578-9158e2736811\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") "
Oct 31 04:07:17.412131 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.412066    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\") pod \"91b45bf1-c805-48ab-8578-9158e2736811\" (UID: \"91b45bf1-c805-48ab-8578-9158e2736811\") "
Oct 31 04:07:17.412392 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.412370    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca" pod "91b45bf1-c805-48ab-8578-9158e2736811" (UID: "91b45bf1-c805-48ab-8578-9158e2736811"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:07:17.412482 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.412466    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:17.414135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.414111    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca" pod "91b45bf1-c805-48ab-8578-9158e2736811" (UID: "91b45bf1-c805-48ab-8578-9158e2736811"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:07:17.414135 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.414123    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r" pod "91b45bf1-c805-48ab-8578-9158e2736811" (UID: "91b45bf1-c805-48ab-8578-9158e2736811"). InnerVolumeSpecName "kube-api-access-gjf8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:17.513756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.513664    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91b45bf1-c805-48ab-8578-9158e2736811-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:17.513756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.513699    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjf8r\" (UniqueName: \"kubernetes.io/projected/91b45bf1-c805-48ab-8578-9158e2736811-kube-api-access-gjf8r\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:17.610083 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:17.610045    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec32b545-8a86-4538-a109-130a12a70cee" path="/var/lib/kubelet/pods/ec32b545-8a86-4538-a109-130a12a70cee/volumes"
Oct 31 04:07:18.204335 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:18.204299    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"
Oct 31 04:07:18.204554 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:18.204339    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h" event={"ID":"91b45bf1-c805-48ab-8578-9158e2736811","Type":"ContainerDied","Data":"158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0"}
Oct 31 04:07:18.204554 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:18.204427    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158bd171b1bd3c0b78e5c0b1197e82c29e7c880a33d324a772387f8aef39ddf0"
Oct 31 04:07:25.260168 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:25.260128    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:07:25.742798 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:25.742743    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 04:07:29.545032 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:29.544996    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:07:30.506802 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.506734    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:07:30.562033 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.561997    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\") pod \"b1220ff4-92fd-41aa-a7f8-65da416dd070\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") "
Oct 31 04:07:30.562529 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.562057    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\" (UniqueName: \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\") pod \"b1220ff4-92fd-41aa-a7f8-65da416dd070\" (UID: \"b1220ff4-92fd-41aa-a7f8-65da416dd070\") "
Oct 31 04:07:30.562529 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.562501    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume" pod "b1220ff4-92fd-41aa-a7f8-65da416dd070" (UID: "b1220ff4-92fd-41aa-a7f8-65da416dd070"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:07:30.564371 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.564337    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8" pod "b1220ff4-92fd-41aa-a7f8-65da416dd070" (UID: "b1220ff4-92fd-41aa-a7f8-65da416dd070"). InnerVolumeSpecName "kube-api-access-q7tq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:07:30.663587 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.663547    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1220ff4-92fd-41aa-a7f8-65da416dd070-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:30.663587 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:30.663585    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tq8\" (UniqueName: \"kubernetes.io/projected/b1220ff4-92fd-41aa-a7f8-65da416dd070-kube-api-access-q7tq8\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:07:31.058636 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:07:31.058583    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:07:31.301905 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.301865    5098 generic.go:356] "Generic (PLEG): container finished" podID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerID="b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61" exitCode=0
Oct 31 04:07:31.302103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.301915    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"
Oct 31 04:07:31.302103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.301927    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" event={"ID":"b1220ff4-92fd-41aa-a7f8-65da416dd070","Type":"ContainerDied","Data":"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"}
Oct 31 04:07:31.302103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.301971    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm" event={"ID":"b1220ff4-92fd-41aa-a7f8-65da416dd070","Type":"ContainerDied","Data":"d2c1638e5b3a8fc923301518febfb6f018e2b2936e0d97fa6ffc9d1f7a7c362e"}
Oct 31 04:07:31.302103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.301992    5098 scope.go:122] "RemoveContainer" containerID="b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"
Oct 31 04:07:31.312597 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.312579    5098 scope.go:122] "RemoveContainer" containerID="b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"
Oct 31 04:07:31.312912 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:07:31.312894    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61\": container with ID starting with b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61 not found: ID does not exist" containerID="b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"
Oct 31 04:07:31.312955 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.312922    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61"} err="failed to get container status \"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61\": rpc error: code = NotFound desc = could not find container \"b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61\": container with ID starting with b3fb81fc996791b5eedd1d967f85f11f1c2de7456f087b1e0dc763e639607d61 not found: ID does not exist"
Oct 31 04:07:31.334589 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.334534    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:07:31.335826 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.335795    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-799962-566455b8bz4dtm"]
Oct 31 04:07:31.611514 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:31.611430    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" path="/var/lib/kubelet/pods/b1220ff4-92fd-41aa-a7f8-65da416dd070/volumes"
Oct 31 04:07:51.021947 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:51.021907    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"]
Oct 31 04:07:51.024358 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:51.024332    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-jph54"]
Oct 31 04:07:51.616047 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:07:51.616006    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b6ebf47-3da1-4e86-9ea1-9e383c783316" path="/var/lib/kubelet/pods/3b6ebf47-3da1-4e86-9ea1-9e383c783316/volumes"
Oct 31 04:08:01.020291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:01.020244    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"]
Oct 31 04:08:01.024134 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:01.024091    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-l95pg"]
Oct 31 04:08:01.057541 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:08:01.057497    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:08:01.611225 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:01.611186    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ab6c736-671e-4b02-999f-b53bf6597b5a" path="/var/lib/kubelet/pods/8ab6c736-671e-4b02-999f-b53bf6597b5a/volumes"
Oct 31 04:08:21.021970 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:21.021931    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"]
Oct 31 04:08:21.024256 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:21.024230    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-shxg6"]
Oct 31 04:08:21.610746 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:21.610708    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10da8251-284b-4078-8d0a-9df7ebae2edb" path="/var/lib/kubelet/pods/10da8251-284b-4078-8d0a-9df7ebae2edb/volumes"
Oct 31 04:08:31.057621 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:08:31.057567    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:08:34.128084 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128047    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"]
Oct 31 04:08:34.128495 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128481    5098 cpu_manager.go:414] "containerMap: removing container" podUID="ec32b545-8a86-4538-a109-130a12a70cee" containerName="main"
Oct 31 04:08:34.128542 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128496    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec32b545-8a86-4538-a109-130a12a70cee" containerName="main"
Oct 31 04:08:34.128542 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128538    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128544    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128552    5098 cpu_manager.go:414] "containerMap: removing container" podUID="91b45bf1-c805-48ab-8578-9158e2736811" containerName="revoke-keys"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128557    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b45bf1-c805-48ab-8578-9158e2736811" containerName="revoke-keys"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128639    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1220ff4-92fd-41aa-a7f8-65da416dd070" containerName="payload-processing"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128647    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b45bf1-c805-48ab-8578-9158e2736811" containerName="revoke-keys"
Oct 31 04:08:34.128658 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.128658    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec32b545-8a86-4538-a109-130a12a70cee" containerName="main"
Oct 31 04:08:34.131994 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.131976    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.137340 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.137317    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-msbqv\""
Oct 31 04:08:34.147837 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.147805    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"]
Oct 31 04:08:34.205030 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.204986    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.205236 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.205105    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nz5\" (UniqueName: \"kubernetes.io/projected/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-kube-api-access-79nz5\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.306103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.306055    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.306324 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.306153    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nz5\" (UniqueName: \"kubernetes.io/projected/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-kube-api-access-79nz5\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.306545 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.306520    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.315557 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.315526    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nz5\" (UniqueName: \"kubernetes.io/projected/6268bbb3-55f4-4c7f-9e77-319f00f7a5a0-kube-api-access-79nz5\") pod \"kuadrant-operator-controller-manager-678d87766f-2h7s4\" (UID: \"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.445462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.445370    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:34.733081 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:34.733056    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"]
Oct 31 04:08:34.735382 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:08:34.735350    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6268bbb3_55f4_4c7f_9e77_319f00f7a5a0.slice/crio-b061a6af8c5ded9f77d53af2ef3500da9fc18f72fcf49b80c18e661a633c255c WatchSource:0}: Error finding container b061a6af8c5ded9f77d53af2ef3500da9fc18f72fcf49b80c18e661a633c255c: Status 404 returned error can't find the container with id b061a6af8c5ded9f77d53af2ef3500da9fc18f72fcf49b80c18e661a633c255c
Oct 31 04:08:35.684435 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:35.684389    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4" event={"ID":"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0","Type":"ContainerStarted","Data":"17ef3b9ee0b9aaa6c649d0fd079033c32289d1d2cd66f6639f109daf9c172040"}
Oct 31 04:08:35.684917 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:35.684464    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:35.684917 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:35.684489    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4" event={"ID":"6268bbb3-55f4-4c7f-9e77-319f00f7a5a0","Type":"ContainerStarted","Data":"b061a6af8c5ded9f77d53af2ef3500da9fc18f72fcf49b80c18e661a633c255c"}
Oct 31 04:08:35.710670 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:35.710594    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4" podStartSLOduration=1.7105803929999999 podStartE2EDuration="1.710580393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:08:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:08:35.708535115 +0000 UTC m=+3190.233754701" watchObservedRunningTime="2026-10-31 04:08:35.710580393 +0000 UTC m=+3190.235799982"
Oct 31 04:08:39.029198 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:39.029155    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"]
Oct 31 04:08:39.031518 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:39.031491    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7kvmk"]
Oct 31 04:08:39.610786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:39.610750    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abbf01e-4330-4aa0-81ff-15694d9b7fc0" path="/var/lib/kubelet/pods/4abbf01e-4330-4aa0-81ff-15694d9b7fc0/volumes"
Oct 31 04:08:46.693712 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:46.693674    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-2h7s4"
Oct 31 04:08:52.451406 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:08:52.451369    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:09:00.021801 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:00.021760    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"]
Oct 31 04:09:00.024201 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:00.024173    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-lffmt"]
Oct 31 04:09:01.056798 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:09:01.056760    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:09:01.610424 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:01.610390    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd372053-0e9e-43da-a696-77584c16abeb" path="/var/lib/kubelet/pods/fd372053-0e9e-43da-a696-77584c16abeb/volumes"
Oct 31 04:09:07.304843 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:07.304804    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:09:11.733560 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:11.733514    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:09:18.022114 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:18.022077    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"]
Oct 31 04:09:18.024776 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:18.024747    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-sjhvz"]
Oct 31 04:09:19.610711 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:19.610673    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717a4a1f-ed88-48c8-88d7-b8009dbdc137" path="/var/lib/kubelet/pods/717a4a1f-ed88-48c8-88d7-b8009dbdc137/volumes"
Oct 31 04:09:25.914410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:25.914373    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:09:31.058233 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:09:31.058185    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:09:34.056933 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:34.056893    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:09:41.350665 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:41.350628    5098 scope.go:122] "RemoveContainer" containerID="47bdcfa6ba2f9e5d013d9752174a2ee87c4de021b56cd14848498895f41d6e66"
Oct 31 04:09:41.364292 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:41.364266    5098 scope.go:122] "RemoveContainer" containerID="e3a3a57745f0a1992d1a5c1c6ff19d867fef3589fb2938b1143881752d5fb165"
Oct 31 04:09:41.373626 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:41.373588    5098 scope.go:122] "RemoveContainer" containerID="1360b59f8726895db7313d632588b765895a5f59e3580008726d48fe594083ff"
Oct 31 04:09:41.382402 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:41.382377    5098 scope.go:122] "RemoveContainer" containerID="95149281258cf490e7ef3cadc6183a19e0f428e4ef4d63e1d764eb096a6643ed"
Oct 31 04:09:48.641529 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:09:48.641487    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:10:01.056552 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:10:01.056509    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:10:06.138685 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:06.138589    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:10:17.741436 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.741399    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"]
Oct 31 04:10:17.745459 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.745435    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.751444 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.751417    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:10:17.751663 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.751414    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:10:17.751780 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.751547    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:10:17.751780 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.751685    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:10:17.753662 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.753637    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"]
Oct 31 04:10:17.827888 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.827849    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.828076 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.827901    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx9rn\" (UniqueName: \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.828076 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.828025    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.929403 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.929361    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.929617 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.929525    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.929617 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.929572    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx9rn\" (UniqueName: \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.930280 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.930257    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.931969 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.931949    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:17.937706 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:17.937677    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx9rn\" (UniqueName: \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\") pod \"maas-api-revoke-sub-77ddec5e-5mj56\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:18.024921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:18.024808    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"]
Oct 31 04:10:18.027202 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:18.027170    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6kkph"]
Oct 31 04:10:18.063600 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:18.063566    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:18.349311 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:18.349279    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"]
Oct 31 04:10:18.350363 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:10:18.350339    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod638decdc_a558_4f3d_829b_42b96c2aa863.slice/crio-0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7 WatchSource:0}: Error finding container 0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7: Status 404 returned error can't find the container with id 0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7
Oct 31 04:10:18.381173 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:18.381142    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56" event={"ID":"638decdc-a558-4f3d-829b-42b96c2aa863","Type":"ContainerStarted","Data":"0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7"}
Oct 31 04:10:19.390114 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:19.390079    5098 generic.go:356] "Generic (PLEG): container finished" podID="638decdc-a558-4f3d-829b-42b96c2aa863" containerID="25bd21e1474e335e3e2e7a91a915e59da928fd2a1514e2e7d76bff58bff95802" exitCode=0
Oct 31 04:10:19.390595 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:19.390175    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56" event={"ID":"638decdc-a558-4f3d-829b-42b96c2aa863","Type":"ContainerDied","Data":"25bd21e1474e335e3e2e7a91a915e59da928fd2a1514e2e7d76bff58bff95802"}
Oct 31 04:10:19.611426 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:19.611389    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dba90cd0-3d85-47e7-beff-c6c191291dcc" path="/var/lib/kubelet/pods/dba90cd0-3d85-47e7-beff-c6c191291dcc/volumes"
Oct 31 04:10:20.552845 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.552813    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:20.658329 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.658229    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\" (UniqueName: \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\") pod \"638decdc-a558-4f3d-829b-42b96c2aa863\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") "
Oct 31 04:10:20.658329 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.658267    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\") pod \"638decdc-a558-4f3d-829b-42b96c2aa863\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") "
Oct 31 04:10:20.658329 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.658325    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\") pod \"638decdc-a558-4f3d-829b-42b96c2aa863\" (UID: \"638decdc-a558-4f3d-829b-42b96c2aa863\") "
Oct 31 04:10:20.658858 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.658823    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca" pod "638decdc-a558-4f3d-829b-42b96c2aa863" (UID: "638decdc-a558-4f3d-829b-42b96c2aa863"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:20.660849 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.660819    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca" pod "638decdc-a558-4f3d-829b-42b96c2aa863" (UID: "638decdc-a558-4f3d-829b-42b96c2aa863"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:20.661067 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.661042    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn" pod "638decdc-a558-4f3d-829b-42b96c2aa863" (UID: "638decdc-a558-4f3d-829b-42b96c2aa863"). InnerVolumeSpecName "kube-api-access-hx9rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:20.759809 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.759762    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx9rn\" (UniqueName: \"kubernetes.io/projected/638decdc-a558-4f3d-829b-42b96c2aa863-kube-api-access-hx9rn\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:20.759809 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.759810    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:20.760032 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:20.759827    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/638decdc-a558-4f3d-829b-42b96c2aa863-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:21.411948 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:21.411914    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"
Oct 31 04:10:21.412141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:21.411939    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56" event={"ID":"638decdc-a558-4f3d-829b-42b96c2aa863","Type":"ContainerDied","Data":"0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7"}
Oct 31 04:10:21.412141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:21.411998    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fae75b8f0a1585ee0c235083b8ea76ed86afa6162b32a54d549e135c79be3c7"
Oct 31 04:10:28.023646 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.023592    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"]
Oct 31 04:10:28.025979 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.025945    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-k7nwg"]
Oct 31 04:10:28.776212 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.776172    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"]
Oct 31 04:10:28.776913 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.776891    5098 cpu_manager.go:414] "containerMap: removing container" podUID="638decdc-a558-4f3d-829b-42b96c2aa863" containerName="revoke-keys"
Oct 31 04:10:28.776974 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.776916    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="638decdc-a558-4f3d-829b-42b96c2aa863" containerName="revoke-keys"
Oct 31 04:10:28.777098 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.777086    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="638decdc-a558-4f3d-829b-42b96c2aa863" containerName="revoke-keys"
Oct 31 04:10:28.780898 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.780876    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.796916 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.796888    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:10:28.797352 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.797236    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:10:28.797503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.797261    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:10:28.797786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.797769    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:10:28.800300 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.800276    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"]
Oct 31 04:10:28.842421 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.842388    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdscm\" (UniqueName: \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.842595 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.842445    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.842595 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.842535    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.858194 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.858160    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:10:28.943939 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.943898    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.944142 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.944037    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdscm\" (UniqueName: \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.944142 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.944094    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.944867 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.944839    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.946866 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.946842    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:28.956468 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:28.956446    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdscm\" (UniqueName: \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\") pod \"maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:29.095104 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:29.095001    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:29.387535 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:29.387509    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"]
Oct 31 04:10:29.389736 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:10:29.389700    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527e09ac_d028_40e6_9524_349e0bc9ed7b.slice/crio-59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8 WatchSource:0}: Error finding container 59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8: Status 404 returned error can't find the container with id 59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8
Oct 31 04:10:29.460007 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:29.459972    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x" event={"ID":"527e09ac-d028-40e6-9524-349e0bc9ed7b","Type":"ContainerStarted","Data":"59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8"}
Oct 31 04:10:29.610562 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:29.610519    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4587127b-c614-4ffc-87c9-a8315881896e" path="/var/lib/kubelet/pods/4587127b-c614-4ffc-87c9-a8315881896e/volumes"
Oct 31 04:10:30.468488 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:30.468387    5098 generic.go:356] "Generic (PLEG): container finished" podID="527e09ac-d028-40e6-9524-349e0bc9ed7b" containerID="db115d2fa758939b1fe3a8aee19ad601fa339fce0ea0ac19bac2f036d39a5f7a" exitCode=0
Oct 31 04:10:30.468488 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:30.468461    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x" event={"ID":"527e09ac-d028-40e6-9524-349e0bc9ed7b","Type":"ContainerDied","Data":"db115d2fa758939b1fe3a8aee19ad601fa339fce0ea0ac19bac2f036d39a5f7a"}
Oct 31 04:10:31.058423 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:10:31.058383    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:10:31.634021 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.631956    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:31.673546 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.673500    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\" (UniqueName: \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\") pod \"527e09ac-d028-40e6-9524-349e0bc9ed7b\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") "
Oct 31 04:10:31.673724 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.673646    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\") pod \"527e09ac-d028-40e6-9524-349e0bc9ed7b\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") "
Oct 31 04:10:31.673786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.673735    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\") pod \"527e09ac-d028-40e6-9524-349e0bc9ed7b\" (UID: \"527e09ac-d028-40e6-9524-349e0bc9ed7b\") "
Oct 31 04:10:31.674134 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.674107    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca" pod "527e09ac-d028-40e6-9524-349e0bc9ed7b" (UID: "527e09ac-d028-40e6-9524-349e0bc9ed7b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:10:31.674288 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.674260    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:31.676403 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.676371    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm" pod "527e09ac-d028-40e6-9524-349e0bc9ed7b" (UID: "527e09ac-d028-40e6-9524-349e0bc9ed7b"). InnerVolumeSpecName "kube-api-access-tdscm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:10:31.676500 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.676374    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca" pod "527e09ac-d028-40e6-9524-349e0bc9ed7b" (UID: "527e09ac-d028-40e6-9524-349e0bc9ed7b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:10:31.775293 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.775185    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdscm\" (UniqueName: \"kubernetes.io/projected/527e09ac-d028-40e6-9524-349e0bc9ed7b-kube-api-access-tdscm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:31.775293 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:31.775225    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/527e09ac-d028-40e6-9524-349e0bc9ed7b-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:10:32.484512 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:32.484457    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x" event={"ID":"527e09ac-d028-40e6-9524-349e0bc9ed7b","Type":"ContainerDied","Data":"59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8"}
Oct 31 04:10:32.484876 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:32.484530    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59adb25ab5ed1477c1cc057fce9a78689f89d4436988c8803c5555a2f2cf71e8"
Oct 31 04:10:32.484876 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:32.484636    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"
Oct 31 04:10:33.070238 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:33.070183    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:10:33.071367 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:33.071343    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:10:40.034428 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:40.034357    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"]
Oct 31 04:10:40.037516 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:40.037486    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-8a336e-p2f7x"]
Oct 31 04:10:41.419029 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:41.418993    5098 scope.go:122] "RemoveContainer" containerID="36cf8b97c2ddf6cc393fe153e9d6937a99d308deeee032330e24d79a4bf7c419"
Oct 31 04:10:41.428492 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:41.428466    5098 scope.go:122] "RemoveContainer" containerID="ad0161e45568a0e4e8a546857c974e49fa8b20ea87837f274e7c6983b726282a"
Oct 31 04:10:41.437257 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:41.437228    5098 scope.go:122] "RemoveContainer" containerID="fd771e3ef51850769d3cf579115073f5156cc20f44cfcd9c8b1c20cc46595538"
Oct 31 04:10:41.610535 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:41.610502    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="527e09ac-d028-40e6-9524-349e0bc9ed7b" path="/var/lib/kubelet/pods/527e09ac-d028-40e6-9524-349e0bc9ed7b/volumes"
Oct 31 04:10:41.810655 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:41.810620    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:10:53.122754 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:53.122716    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:10:56.144814 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:56.144771    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:10:56.145297 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:56.144847    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:10:56.162643 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:56.161708    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:10:56.162643 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:10:56.161800    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:11:01.057005 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:11:01.056967    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:11:09.131396 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:09.131355    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:11:24.769290 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.769258    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"]
Oct 31 04:11:24.769794 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.769755    5098 cpu_manager.go:414] "containerMap: removing container" podUID="527e09ac-d028-40e6-9524-349e0bc9ed7b" containerName="revoke-keys"
Oct 31 04:11:24.769859 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.769797    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="527e09ac-d028-40e6-9524-349e0bc9ed7b" containerName="revoke-keys"
Oct 31 04:11:24.769950 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.769934    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="527e09ac-d028-40e6-9524-349e0bc9ed7b" containerName="revoke-keys"
Oct 31 04:11:24.773615 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.773585    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.779137 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.779108    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:11:24.779436 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.779410    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:11:24.779436 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.779120    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:11:24.779671 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.779129    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:11:24.782903 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.782882    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"]
Oct 31 04:11:24.874983 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.874949    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.875181 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.875003    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.875181 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.875104    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnmm\" (UniqueName: \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.976097 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.976060    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.976345 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.976127    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.976345 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.976169    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnmm\" (UniqueName: \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.976976 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.976947    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.978753 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.978730    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:24.984681 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:24.984659    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnmm\" (UniqueName: \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\") pod \"maas-api-revoke-sub-452b6d7b-kgcr5\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:25.089423 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.089373    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:25.369890 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.369859    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"]
Oct 31 04:11:25.371438 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:11:25.371404    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049a3de7_75ee_4fb1_a321_945677422802.slice/crio-8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b WatchSource:0}: Error finding container 8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b: Status 404 returned error can't find the container with id 8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b
Oct 31 04:11:25.405558 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.405527    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:11:25.827418 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.827373    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5" event={"ID":"049a3de7-75ee-4fb1-a321-945677422802","Type":"ContainerStarted","Data":"02713c10e746c866bab5e03daea00302b245a3e27d75a8dbe743610ff36c1f74"}
Oct 31 04:11:25.827986 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.827449    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5" event={"ID":"049a3de7-75ee-4fb1-a321-945677422802","Type":"ContainerStarted","Data":"8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b"}
Oct 31 04:11:25.864002 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:25.863931    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5" podStartSLOduration=1.863909403 podStartE2EDuration="1.863909403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:25.863142147 +0000 UTC m=+3360.388361736" watchObservedRunningTime="2026-10-31 04:11:25.863909403 +0000 UTC m=+3360.389128993"
Oct 31 04:11:26.839214 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:26.839169    5098 generic.go:356] "Generic (PLEG): container finished" podID="049a3de7-75ee-4fb1-a321-945677422802" containerID="02713c10e746c866bab5e03daea00302b245a3e27d75a8dbe743610ff36c1f74" exitCode=0
Oct 31 04:11:26.839911 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:26.839215    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5" event={"ID":"049a3de7-75ee-4fb1-a321-945677422802","Type":"ContainerDied","Data":"02713c10e746c866bab5e03daea00302b245a3e27d75a8dbe743610ff36c1f74"}
Oct 31 04:11:28.412955 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.412858    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:28.564632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.550008    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\") pod \"049a3de7-75ee-4fb1-a321-945677422802\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") "
Oct 31 04:11:28.564632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.550069    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\" (UniqueName: \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\") pod \"049a3de7-75ee-4fb1-a321-945677422802\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") "
Oct 31 04:11:28.564632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.550185    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\") pod \"049a3de7-75ee-4fb1-a321-945677422802\" (UID: \"049a3de7-75ee-4fb1-a321-945677422802\") "
Oct 31 04:11:28.569587 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.569307    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca" pod "049a3de7-75ee-4fb1-a321-945677422802" (UID: "049a3de7-75ee-4fb1-a321-945677422802"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:28.570243 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.569917    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca" pod "049a3de7-75ee-4fb1-a321-945677422802" (UID: "049a3de7-75ee-4fb1-a321-945677422802"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:28.587562 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.587501    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm" pod "049a3de7-75ee-4fb1-a321-945677422802" (UID: "049a3de7-75ee-4fb1-a321-945677422802"). InnerVolumeSpecName "kube-api-access-gfnmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:28.660632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.658304    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/049a3de7-75ee-4fb1-a321-945677422802-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:28.660632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.658352    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/049a3de7-75ee-4fb1-a321-945677422802-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:28.660632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.658372    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnmm\" (UniqueName: \"kubernetes.io/projected/049a3de7-75ee-4fb1-a321-945677422802-kube-api-access-gfnmm\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:28.864632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.862505    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5" event={"ID":"049a3de7-75ee-4fb1-a321-945677422802","Type":"ContainerDied","Data":"8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b"}
Oct 31 04:11:28.864632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.862576    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8723e5df0b222a7d012cb71bd7050f94203ec8c995dc569437dc12dc4ade219b"
Oct 31 04:11:28.864632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:28.862664    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"
Oct 31 04:11:31.059793 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:11:31.059755    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:11:36.427882 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:36.427760    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:11:37.788551 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.788510    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"]
Oct 31 04:11:37.789202 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.789183    5098 cpu_manager.go:414] "containerMap: removing container" podUID="049a3de7-75ee-4fb1-a321-945677422802" containerName="revoke-keys"
Oct 31 04:11:37.789253 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.789204    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="049a3de7-75ee-4fb1-a321-945677422802" containerName="revoke-keys"
Oct 31 04:11:37.789335 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.789323    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="049a3de7-75ee-4fb1-a321-945677422802" containerName="revoke-keys"
Oct 31 04:11:37.792839 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.792815    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:37.800653 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.798861    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:11:37.800653 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.799082    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:11:37.800653 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.799082    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:11:37.800653 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.799204    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:11:37.801471 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.801439    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"]
Oct 31 04:11:37.989513 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.989472    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:37.989746 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.989666    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:37.989746 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:37.989710    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8827\" (UniqueName: \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.091094 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.090988    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.091094 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.091068    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.091094 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.091097    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8827\" (UniqueName: \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.091778 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.091751    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.093619 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.093578    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.099757 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.099729    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8827\" (UniqueName: \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\") pod \"maas-api-revoke-sub-3e31a506-rl8hs\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.113506 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.113476    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:38.389712 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.389677    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"]
Oct 31 04:11:38.391317 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:11:38.391282    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8028d51f_aab9_40b7_836a_18bbd7c4c8d1.slice/crio-f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de WatchSource:0}: Error finding container f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de: Status 404 returned error can't find the container with id f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de
Oct 31 04:11:38.926061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.925956    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs" event={"ID":"8028d51f-aab9-40b7-836a-18bbd7c4c8d1","Type":"ContainerStarted","Data":"704fd4ba0ff9bd8295b71795dedab4fd98d34f874b2872946c63a5623d7a6e39"}
Oct 31 04:11:38.926061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.926026    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs" event={"ID":"8028d51f-aab9-40b7-836a-18bbd7c4c8d1","Type":"ContainerStarted","Data":"f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de"}
Oct 31 04:11:38.944058 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:38.943999    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs" podStartSLOduration=1.943986099 podStartE2EDuration="1.943986099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:11:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:11:38.941908774 +0000 UTC m=+3373.467128387" watchObservedRunningTime="2026-10-31 04:11:38.943986099 +0000 UTC m=+3373.469205687"
Oct 31 04:11:39.937778 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:39.937735    5098 generic.go:356] "Generic (PLEG): container finished" podID="8028d51f-aab9-40b7-836a-18bbd7c4c8d1" containerID="704fd4ba0ff9bd8295b71795dedab4fd98d34f874b2872946c63a5623d7a6e39" exitCode=0
Oct 31 04:11:39.938382 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:39.937802    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs" event={"ID":"8028d51f-aab9-40b7-836a-18bbd7c4c8d1","Type":"ContainerDied","Data":"704fd4ba0ff9bd8295b71795dedab4fd98d34f874b2872946c63a5623d7a6e39"}
Oct 31 04:11:41.545420 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.540387    5098 scope.go:122] "RemoveContainer" containerID="450932934dbfdb8434167fe81dc3554d0a22cd95f92700ef2074e5eaf010f562"
Oct 31 04:11:41.624635 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.602050    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:41.630310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.630281    5098 scope.go:122] "RemoveContainer" containerID="65dc84897fbdaefc772ee0df49d55475d92c67c29b0f70ae9305427336b1a0aa"
Oct 31 04:11:41.639193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.639149    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:11:41.681656 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.658903    5098 scope.go:122] "RemoveContainer" containerID="39c452de1ca2a9c5762f16c8bf57855edf67b0011c5acc03c8920fb5e77cb9df"
Oct 31 04:11:41.727773 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.727656    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\" (UniqueName: \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\") pod \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") "
Oct 31 04:11:41.728098 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.728073    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\") pod \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") "
Oct 31 04:11:41.729707 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.729371    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\") pod \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\" (UID: \"8028d51f-aab9-40b7-836a-18bbd7c4c8d1\") "
Oct 31 04:11:41.729707 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.728417    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca" pod "8028d51f-aab9-40b7-836a-18bbd7c4c8d1" (UID: "8028d51f-aab9-40b7-836a-18bbd7c4c8d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:11:41.735706 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.730003    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:41.759977 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.759914    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827" pod "8028d51f-aab9-40b7-836a-18bbd7c4c8d1" (UID: "8028d51f-aab9-40b7-836a-18bbd7c4c8d1"). InnerVolumeSpecName "kube-api-access-x8827". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:11:41.805818 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.805754    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca" pod "8028d51f-aab9-40b7-836a-18bbd7c4c8d1" (UID: "8028d51f-aab9-40b7-836a-18bbd7c4c8d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:11:41.841331 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.834355    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:41.841331 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.834403    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8827\" (UniqueName: \"kubernetes.io/projected/8028d51f-aab9-40b7-836a-18bbd7c4c8d1-kube-api-access-x8827\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:11:41.847895 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:41.847860    5098 scope.go:122] "RemoveContainer" containerID="d6a39d864cd52fbd47a749bc20f14d6e78d846790094a129e808eb0abf693384"
Oct 31 04:11:42.048805 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:42.048762    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs" event={"ID":"8028d51f-aab9-40b7-836a-18bbd7c4c8d1","Type":"ContainerDied","Data":"f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de"}
Oct 31 04:11:42.049123 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:42.049099    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f48bbd4d4496d98330dd814983e65e48139b0bd0b7c9a620bafd5336f04173de"
Oct 31 04:11:42.049348 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:42.049334    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"
Oct 31 04:11:48.644555 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:48.644514    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:11:53.187632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:11:53.186967    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:12:01.056031 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:01.055993    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:12:08.926643 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:08.926581    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:12:10.276961 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.276881    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:10.277752 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.277717    5098 cpu_manager.go:414] "containerMap: removing container" podUID="8028d51f-aab9-40b7-836a-18bbd7c4c8d1" containerName="revoke-keys"
Oct 31 04:12:10.277752 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.277743    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8028d51f-aab9-40b7-836a-18bbd7c4c8d1" containerName="revoke-keys"
Oct 31 04:12:10.278044 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.277908    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="8028d51f-aab9-40b7-836a-18bbd7c4c8d1" containerName="revoke-keys"
Oct 31 04:12:10.281871 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.281849    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.292389 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.292362    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:10.292740 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.292712    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 04:12:10.293286 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.293085    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 04:12:10.293286 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.293153    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-dbwgh\""
Oct 31 04:12:10.320828 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.320787    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:12:10.321031 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.320895    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:12:10.401593 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.401549    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.401798 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.401598    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.401798 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.401727    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.401902 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.401891    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45rf\" (UniqueName: \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.502867 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.502823    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45rf\" (UniqueName: \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.503091 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.502908    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.503091 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.502934    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.503091 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.502964    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.503648 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.503601    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.505462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.505434    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.505593 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.505575    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.511570 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.511544    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45rf\" (UniqueName: \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\") pod \"maas-api-78fdcdcc4-sxl6h\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") " pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.602912 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.602817    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:10.881377 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.881344    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:10.882843 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:12:10.882816    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb41226e5_3609_4f9e_a7be_cec9678a631e.slice/crio-62569fd70ace68a053176108580bd30d68a76d3eb511596a8582be08b64a28a6 WatchSource:0}: Error finding container 62569fd70ace68a053176108580bd30d68a76d3eb511596a8582be08b64a28a6: Status 404 returned error can't find the container with id 62569fd70ace68a053176108580bd30d68a76d3eb511596a8582be08b64a28a6
Oct 31 04:12:10.884380 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:10.884359    5098 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 04:12:11.344410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:11.344371    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" event={"ID":"b41226e5-3609-4f9e-a7be-cec9678a631e","Type":"ContainerStarted","Data":"62569fd70ace68a053176108580bd30d68a76d3eb511596a8582be08b64a28a6"}
Oct 31 04:12:12.351242 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:12.351193    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" event={"ID":"b41226e5-3609-4f9e-a7be-cec9678a631e","Type":"ContainerStarted","Data":"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"}
Oct 31 04:12:12.351729 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:12.351299    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:12.371746 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:12.371678    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" podStartSLOduration=1.783197511 podStartE2EDuration="2.371661738s" totalImagesPullingTime="588.464227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:12.368646953 +0000 UTC m=+3406.893866545" watchObservedRunningTime="2026-10-31 04:12:12.371661738 +0000 UTC m=+3406.896881371"
Oct 31 04:12:17.727051 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:17.727006    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:17.729600 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:17.727315    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" podUID="b41226e5-3609-4f9e-a7be-cec9678a631e" containerName="maas-api" containerID="cri-o://5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980" gracePeriod=30
Oct 31 04:12:17.734241 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:17.734213    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:17.993629 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:17.993552    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:18.026547 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.026508    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"]
Oct 31 04:12:18.033741 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.033706    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-64p2h"]
Oct 31 04:12:18.080863 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.080823    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\") pod \"b41226e5-3609-4f9e-a7be-cec9678a631e\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") "
Oct 31 04:12:18.081084 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.080909    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\") pod \"b41226e5-3609-4f9e-a7be-cec9678a631e\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") "
Oct 31 04:12:18.081084 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.080935    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\" (UniqueName: \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\") pod \"b41226e5-3609-4f9e-a7be-cec9678a631e\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") "
Oct 31 04:12:18.081084 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.080980    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\") pod \"b41226e5-3609-4f9e-a7be-cec9678a631e\" (UID: \"b41226e5-3609-4f9e-a7be-cec9678a631e\") "
Oct 31 04:12:18.081446 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.081412    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca" pod "b41226e5-3609-4f9e-a7be-cec9678a631e" (UID: "b41226e5-3609-4f9e-a7be-cec9678a631e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:12:18.083147 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.083122    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls" pod "b41226e5-3609-4f9e-a7be-cec9678a631e" (UID: "b41226e5-3609-4f9e-a7be-cec9678a631e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:12:18.083247 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.083204    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf" pod "b41226e5-3609-4f9e-a7be-cec9678a631e" (UID: "b41226e5-3609-4f9e-a7be-cec9678a631e"). InnerVolumeSpecName "kube-api-access-v45rf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:12:18.083337 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.083312    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs" pod "b41226e5-3609-4f9e-a7be-cec9678a631e" (UID: "b41226e5-3609-4f9e-a7be-cec9678a631e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:12:18.182410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.182361    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-maas-api-tls\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:18.182410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.182404    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45rf\" (UniqueName: \"kubernetes.io/projected/b41226e5-3609-4f9e-a7be-cec9678a631e-kube-api-access-v45rf\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:18.182410 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.182420    5098 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b41226e5-3609-4f9e-a7be-cec9678a631e-openshift-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:18.182721 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.182432    5098 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b41226e5-3609-4f9e-a7be-cec9678a631e-metrics-certs\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:18.386842 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.386805    5098 generic.go:356] "Generic (PLEG): container finished" podID="b41226e5-3609-4f9e-a7be-cec9678a631e" containerID="5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980" exitCode=0
Oct 31 04:12:18.387035 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.386863    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"
Oct 31 04:12:18.387035 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.386888    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" event={"ID":"b41226e5-3609-4f9e-a7be-cec9678a631e","Type":"ContainerDied","Data":"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"}
Oct 31 04:12:18.387035 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.386960    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h" event={"ID":"b41226e5-3609-4f9e-a7be-cec9678a631e","Type":"ContainerDied","Data":"62569fd70ace68a053176108580bd30d68a76d3eb511596a8582be08b64a28a6"}
Oct 31 04:12:18.387035 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.386980    5098 scope.go:122] "RemoveContainer" containerID="5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"
Oct 31 04:12:18.401717 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.401694    5098 scope.go:122] "RemoveContainer" containerID="5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"
Oct 31 04:12:18.402029 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:18.402008    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980\": container with ID starting with 5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980 not found: ID does not exist" containerID="5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"
Oct 31 04:12:18.402106 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.402040    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980"} err="failed to get container status \"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980\": rpc error: code = NotFound desc = could not find container \"5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980\": container with ID starting with 5b1e829198503579750ff69ad222a74fe52549ca879d97db67b1da65e2c06980 not found: ID does not exist"
Oct 31 04:12:18.411994 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.411966    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:18.414163 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:18.414136    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78fdcdcc4-sxl6h"]
Oct 31 04:12:19.610906 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:19.610869    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b45bf1-c805-48ab-8578-9158e2736811" path="/var/lib/kubelet/pods/91b45bf1-c805-48ab-8578-9158e2736811/volumes"
Oct 31 04:12:19.611435 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:19.611417    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41226e5-3609-4f9e-a7be-cec9678a631e" path="/var/lib/kubelet/pods/b41226e5-3609-4f9e-a7be-cec9678a631e/volumes"
Oct 31 04:12:21.455716 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.455672    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:21.456164 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.456149    5098 cpu_manager.go:414] "containerMap: removing container" podUID="b41226e5-3609-4f9e-a7be-cec9678a631e" containerName="maas-api"
Oct 31 04:12:21.456210 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.456166    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41226e5-3609-4f9e-a7be-cec9678a631e" containerName="maas-api"
Oct 31 04:12:21.456262 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.456253    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41226e5-3609-4f9e-a7be-cec9678a631e" containerName="maas-api"
Oct 31 04:12:21.460949 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.460928    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.469356 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.468860    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:21.469356 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.468918    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4bd78aa5\""
Oct 31 04:12:21.476947 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.476913    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:21.481039 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.481012    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.485895 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.485855    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4bd78aa5-dockercfg-b9mnw\""
Oct 31 04:12:21.490038 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.490003    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:21.618516 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.618468    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplk9\" (UniqueName: \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.618727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.618570    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.618727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.618687    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn7v\" (UniqueName: \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.618727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.618719    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.720291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.720195    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplk9\" (UniqueName: \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.720291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.720246    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.720291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.720289    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn7v\" (UniqueName: \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.720599 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.720304    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.721192 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.721164    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.721263 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.721174    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.729260 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.729231    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn7v\" (UniqueName: \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\") pod \"payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:21.729379 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.729260    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplk9\" (UniqueName: \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\") pod \"payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") " pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.779057 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.779020    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:21.804427 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:21.804390    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:22.077367 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.077323    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:22.126968 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.126937    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:22.128261 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:12:22.128234    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea21751_2bda_40e5_9d61_57e9b461067b.slice/crio-7d9cf2689cae8e03a9ce946c096878e395119696f0fd260b01483b1178375b85 WatchSource:0}: Error finding container 7d9cf2689cae8e03a9ce946c096878e395119696f0fd260b01483b1178375b85: Status 404 returned error can't find the container with id 7d9cf2689cae8e03a9ce946c096878e395119696f0fd260b01483b1178375b85
Oct 31 04:12:22.414966 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.414850    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" event={"ID":"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58","Type":"ContainerStarted","Data":"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"}
Oct 31 04:12:22.414966 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.414940    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:22.415223 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.414966    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" event={"ID":"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58","Type":"ContainerStarted","Data":"8fb705a036c88e79c6a80fb5293030a3561c5f9228ab21e9d9d9bc64a7dea85b"}
Oct 31 04:12:22.416437 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.416407    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" event={"ID":"dea21751-2bda-40e5-9d61-57e9b461067b","Type":"ContainerStarted","Data":"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"}
Oct 31 04:12:22.416562 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.416459    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" event={"ID":"dea21751-2bda-40e5-9d61-57e9b461067b","Type":"ContainerStarted","Data":"7d9cf2689cae8e03a9ce946c096878e395119696f0fd260b01483b1178375b85"}
Oct 31 04:12:22.416562 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.416498    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:22.432971 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.432904    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" podStartSLOduration=1.432883359 podStartE2EDuration="1.432883359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:22.431735795 +0000 UTC m=+3416.956955387" watchObservedRunningTime="2026-10-31 04:12:22.432883359 +0000 UTC m=+3416.958102957"
Oct 31 04:12:22.450996 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:22.450945    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" podStartSLOduration=1.450929177 podStartE2EDuration="1.450929177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:22.449035547 +0000 UTC m=+3416.974255152" watchObservedRunningTime="2026-10-31 04:12:22.450929177 +0000 UTC m=+3416.976148766"
Oct 31 04:12:23.429256 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:23.429220    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:12:30.795107 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.795068    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"]
Oct 31 04:12:30.799091 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.799070    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:30.804872 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.804837    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-tfdls\""
Oct 31 04:12:30.804872 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.804865    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:12:30.805148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.804958    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:12:30.805148 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.804969    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:12:30.806894 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.806868    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"]
Oct 31 04:12:30.918559 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.918525    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:30.918776 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.918577    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:30.918828 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:30.918771    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.019351 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.019312    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.019558 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.019360    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.019558 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.019439    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.020243 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.020215    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.021999 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.021964    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.025489 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:31.025462    5098 projected.go:196] Error preparing data for projected volume kube-api-access-jrngv for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 04:12:31.025624 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:31.025544    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv podName:62285e76-2c3a-4f15-abeb-7eec420a86ce nodeName:}" failed. No retries permitted until 2026-10-31 04:12:31.525528921 +0000 UTC m=+3426.050748492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-jrngv" (UniqueName: "kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv") pod "maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" (UID: "62285e76-2c3a-4f15-abeb-7eec420a86ce") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 04:12:31.057889 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:31.057796    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:12:31.626232 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:31.626195    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:31.630523 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:31.630501    5098 projected.go:196] Error preparing data for projected volume kube-api-access-jrngv for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 04:12:31.630689 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:31.630572    5098 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv podName:62285e76-2c3a-4f15-abeb-7eec420a86ce nodeName:}" failed. No retries permitted until 2026-10-31 04:12:32.630549839 +0000 UTC m=+3427.155769428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-jrngv" (UniqueName: "kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv") pod "maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" (UID: "62285e76-2c3a-4f15-abeb-7eec420a86ce") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Oct 31 04:12:32.636178 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:32.636136    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:32.645231 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:32.645195    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:32.917414 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:32.917282    5098 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" secret="" err="secret \"maas-api-cleanup-dockercfg-tfdls\" not found"
Oct 31 04:12:32.917414 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:32.917350    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:33.202786 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:33.202752    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"]
Oct 31 04:12:33.204060 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:12:33.204025    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62285e76_2c3a_4f15_abeb_7eec420a86ce.slice/crio-8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef WatchSource:0}: Error finding container 8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef: Status 404 returned error can't find the container with id 8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef
Oct 31 04:12:33.425304 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:33.425269    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:33.425495 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:33.425336    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:33.498976 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:33.498921    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" event={"ID":"62285e76-2c3a-4f15-abeb-7eec420a86ce","Type":"ContainerStarted","Data":"8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef"}
Oct 31 04:12:34.506784 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:34.506680    5098 generic.go:356] "Generic (PLEG): container finished" podID="62285e76-2c3a-4f15-abeb-7eec420a86ce" containerID="feef3ac926806e139451d60957977276893dffb412e22d6dfe29ee51dd90833b" exitCode=0
Oct 31 04:12:34.506784 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:34.506729    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" event={"ID":"62285e76-2c3a-4f15-abeb-7eec420a86ce","Type":"ContainerDied","Data":"feef3ac926806e139451d60957977276893dffb412e22d6dfe29ee51dd90833b"}
Oct 31 04:12:35.650923 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.650888    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:35.767088 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.766984    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") pod \"62285e76-2c3a-4f15-abeb-7eec420a86ce\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") "
Oct 31 04:12:35.767088 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.767050    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\") pod \"62285e76-2c3a-4f15-abeb-7eec420a86ce\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") "
Oct 31 04:12:35.767294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.767126    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\") pod \"62285e76-2c3a-4f15-abeb-7eec420a86ce\" (UID: \"62285e76-2c3a-4f15-abeb-7eec420a86ce\") "
Oct 31 04:12:35.767475 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.767448    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca" pod "62285e76-2c3a-4f15-abeb-7eec420a86ce" (UID: "62285e76-2c3a-4f15-abeb-7eec420a86ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:12:35.769339 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.769300    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca" pod "62285e76-2c3a-4f15-abeb-7eec420a86ce" (UID: "62285e76-2c3a-4f15-abeb-7eec420a86ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:12:35.769520 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.769426    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv" pod "62285e76-2c3a-4f15-abeb-7eec420a86ce" (UID: "62285e76-2c3a-4f15-abeb-7eec420a86ce"). InnerVolumeSpecName "kube-api-access-jrngv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:12:35.868341 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.868301    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrngv\" (UniqueName: \"kubernetes.io/projected/62285e76-2c3a-4f15-abeb-7eec420a86ce-kube-api-access-jrngv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:35.868341 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.868336    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:35.868341 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:35.868347    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62285e76-2c3a-4f15-abeb-7eec420a86ce-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:36.520714 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:36.520685    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"
Oct 31 04:12:36.520714 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:36.520701    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75" event={"ID":"62285e76-2c3a-4f15-abeb-7eec420a86ce","Type":"ContainerDied","Data":"8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef"}
Oct 31 04:12:36.520955 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:36.520756    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8624d150c61805fa5c70424c26d55bfdd5980c655a70d54c4f16e53a3cb820ef"
Oct 31 04:12:40.929152 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:40.929112    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:40.944697 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:40.944662    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:41.557492 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:41.557455    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"]
Oct 31 04:12:41.562851 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:41.562813    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-4bd78aa5-99w75"]
Oct 31 04:12:41.610503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:41.610470    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62285e76-2c3a-4f15-abeb-7eec420a86ce" path="/var/lib/kubelet/pods/62285e76-2c3a-4f15-abeb-7eec420a86ce/volumes"
Oct 31 04:12:42.841997 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:42.841954    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:12:45.930491 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:45.930447    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" podUID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" containerName="payload-processing" containerID="cri-o://d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900" gracePeriod=25
Oct 31 04:12:45.945162 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:45.945124    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" podUID="dea21751-2bda-40e5-9d61-57e9b461067b" containerName="payload-pre-processing" containerID="cri-o://ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396" gracePeriod=25
Oct 31 04:12:46.220517 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.220489    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:46.259994 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.259961    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:46.271092 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.271060    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\") pod \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") "
Oct 31 04:12:46.271261 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.271107    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\" (UniqueName: \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\") pod \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\" (UID: \"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58\") "
Oct 31 04:12:46.271448 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.271421    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume" pod "4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" (UID: "4ba7dab5-e9e3-42e6-bb88-72a5854fbc58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:12:46.273302 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.273280    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9" pod "4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" (UID: "4ba7dab5-e9e3-42e6-bb88-72a5854fbc58"). InnerVolumeSpecName "kube-api-access-gplk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:12:46.372337 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.372302    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\" (UniqueName: \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\") pod \"dea21751-2bda-40e5-9d61-57e9b461067b\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") "
Oct 31 04:12:46.372529 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.372397    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\") pod \"dea21751-2bda-40e5-9d61-57e9b461067b\" (UID: \"dea21751-2bda-40e5-9d61-57e9b461067b\") "
Oct 31 04:12:46.372711 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.372695    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:46.372763 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.372712    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplk9\" (UniqueName: \"kubernetes.io/projected/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58-kube-api-access-gplk9\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:46.372865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.372837    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume" pod "dea21751-2bda-40e5-9d61-57e9b461067b" (UID: "dea21751-2bda-40e5-9d61-57e9b461067b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:12:46.374682 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.374657    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v" pod "dea21751-2bda-40e5-9d61-57e9b461067b" (UID: "dea21751-2bda-40e5-9d61-57e9b461067b"). InnerVolumeSpecName "kube-api-access-xdn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:12:46.474335 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.474244    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdn7v\" (UniqueName: \"kubernetes.io/projected/dea21751-2bda-40e5-9d61-57e9b461067b-kube-api-access-xdn7v\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:46.474335 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.474281    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dea21751-2bda-40e5-9d61-57e9b461067b-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:12:46.581998 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.581955    5098 generic.go:356] "Generic (PLEG): container finished" podID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" containerID="d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900" exitCode=0
Oct 31 04:12:46.582188 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.582030    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"
Oct 31 04:12:46.582188 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.582038    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" event={"ID":"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58","Type":"ContainerDied","Data":"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"}
Oct 31 04:12:46.582188 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.582123    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz" event={"ID":"4ba7dab5-e9e3-42e6-bb88-72a5854fbc58","Type":"ContainerDied","Data":"8fb705a036c88e79c6a80fb5293030a3561c5f9228ab21e9d9d9bc64a7dea85b"}
Oct 31 04:12:46.582188 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.582137    5098 scope.go:122] "RemoveContainer" containerID="d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"
Oct 31 04:12:46.583685 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.583649    5098 generic.go:356] "Generic (PLEG): container finished" podID="dea21751-2bda-40e5-9d61-57e9b461067b" containerID="ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396" exitCode=0
Oct 31 04:12:46.583821 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.583715    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"
Oct 31 04:12:46.583821 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.583729    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" event={"ID":"dea21751-2bda-40e5-9d61-57e9b461067b","Type":"ContainerDied","Data":"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"}
Oct 31 04:12:46.583821 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.583774    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj" event={"ID":"dea21751-2bda-40e5-9d61-57e9b461067b","Type":"ContainerDied","Data":"7d9cf2689cae8e03a9ce946c096878e395119696f0fd260b01483b1178375b85"}
Oct 31 04:12:46.592705 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.592682    5098 scope.go:122] "RemoveContainer" containerID="d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"
Oct 31 04:12:46.593070 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:46.593036    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900\": container with ID starting with d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900 not found: ID does not exist" containerID="d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"
Oct 31 04:12:46.593194 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.593082    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900"} err="failed to get container status \"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900\": rpc error: code = NotFound desc = could not find container \"d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900\": container with ID starting with d6394432452c60b7097cfb151ea6b9e2d56db8c18e39a91073fc8e5a243f4900 not found: ID does not exist"
Oct 31 04:12:46.593194 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.593110    5098 scope.go:122] "RemoveContainer" containerID="ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"
Oct 31 04:12:46.602700 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.602678    5098 scope.go:122] "RemoveContainer" containerID="ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"
Oct 31 04:12:46.603024 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:12:46.603004    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396\": container with ID starting with ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396 not found: ID does not exist" containerID="ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"
Oct 31 04:12:46.603080 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.603034    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396"} err="failed to get container status \"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396\": rpc error: code = NotFound desc = could not find container \"ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396\": container with ID starting with ae967302961c546c5d918cf22e9e5218e0db61dddd0f0e229384a519cdb78396 not found: ID does not exist"
Oct 31 04:12:46.611169 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.611138    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:46.615359 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.615332    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-4bd78aa5-6bd5848d55-v26kj"]
Oct 31 04:12:46.626571 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.626525    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:46.628849 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:46.628825    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4bd78aa5-566c76cddc-mb6lz"]
Oct 31 04:12:47.611244 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:47.611205    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" path="/var/lib/kubelet/pods/4ba7dab5-e9e3-42e6-bb88-72a5854fbc58/volumes"
Oct 31 04:12:47.612070 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:47.612050    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea21751-2bda-40e5-9d61-57e9b461067b" path="/var/lib/kubelet/pods/dea21751-2bda-40e5-9d61-57e9b461067b/volumes"
Oct 31 04:12:53.285865 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.285827    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:12:53.286436 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286401    5098 cpu_manager.go:414] "containerMap: removing container" podUID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" containerName="payload-processing"
Oct 31 04:12:53.286436 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286420    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" containerName="payload-processing"
Oct 31 04:12:53.286540 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286465    5098 cpu_manager.go:414] "containerMap: removing container" podUID="62285e76-2c3a-4f15-abeb-7eec420a86ce" containerName="revoke-keys"
Oct 31 04:12:53.286540 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286475    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62285e76-2c3a-4f15-abeb-7eec420a86ce" containerName="revoke-keys"
Oct 31 04:12:53.286540 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286504    5098 cpu_manager.go:414] "containerMap: removing container" podUID="dea21751-2bda-40e5-9d61-57e9b461067b" containerName="payload-pre-processing"
Oct 31 04:12:53.286540 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286513    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea21751-2bda-40e5-9d61-57e9b461067b" containerName="payload-pre-processing"
Oct 31 04:12:53.286752 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286626    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="62285e76-2c3a-4f15-abeb-7eec420a86ce" containerName="revoke-keys"
Oct 31 04:12:53.286752 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286641    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea21751-2bda-40e5-9d61-57e9b461067b" containerName="payload-pre-processing"
Oct 31 04:12:53.286752 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.286648    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ba7dab5-e9e3-42e6-bb88-72a5854fbc58" containerName="payload-processing"
Oct 31 04:12:53.291620 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.291580    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.299964 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.299928    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:12:53.300394 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.300197    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-5e424f\""
Oct 31 04:12:53.321765 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.321732    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-5e424f-dockercfg-zlrgk\""
Oct 31 04:12:53.339848 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.339815    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:12:53.342883 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.342852    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.343029 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.342922    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsfv\" (UniqueName: \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.343775 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.343759    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.353103 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.353072    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:12:53.444777 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.444734    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.445043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.444794    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k5nk\" (UniqueName: \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.445043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.444851    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.445043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.444967    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsfv\" (UniqueName: \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.446249 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.446216    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.456723 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.456691    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsfv\" (UniqueName: \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\") pod \"payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.546397 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.546293    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k5nk\" (UniqueName: \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.546397 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.546352    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.547134 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.547113    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.555795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.555769    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k5nk\" (UniqueName: \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\") pod \"payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.614545 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.614514    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:53.660141 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.660006    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:53.900107 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.900073    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:12:53.901688 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:12:53.901656    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db32cbf_918a_4483_8601_322010cfd9cc.slice/crio-d7798cfc36449647afa5d59b0d2891c476b6000d9b450443d6b12d1b833cac71 WatchSource:0}: Error finding container d7798cfc36449647afa5d59b0d2891c476b6000d9b450443d6b12d1b833cac71: Status 404 returned error can't find the container with id d7798cfc36449647afa5d59b0d2891c476b6000d9b450443d6b12d1b833cac71
Oct 31 04:12:53.981804 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:53.981591    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:12:53.984755 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:12:53.984723    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdcac7b_1332_4550_aab2_4a31878c564d.slice/crio-8ba32589af71f45f30008b433850e8fbc74a3aa813e9c2880272a51e48d5d3aa WatchSource:0}: Error finding container 8ba32589af71f45f30008b433850e8fbc74a3aa813e9c2880272a51e48d5d3aa: Status 404 returned error can't find the container with id 8ba32589af71f45f30008b433850e8fbc74a3aa813e9c2880272a51e48d5d3aa
Oct 31 04:12:54.647932 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.647889    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" event={"ID":"9db32cbf-918a-4483-8601-322010cfd9cc","Type":"ContainerStarted","Data":"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"}
Oct 31 04:12:54.648412 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.647965    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:12:54.648412 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.647989    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" event={"ID":"9db32cbf-918a-4483-8601-322010cfd9cc","Type":"ContainerStarted","Data":"d7798cfc36449647afa5d59b0d2891c476b6000d9b450443d6b12d1b833cac71"}
Oct 31 04:12:54.649663 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.649632    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" event={"ID":"2fdcac7b-1332-4550-aab2-4a31878c564d","Type":"ContainerStarted","Data":"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"}
Oct 31 04:12:54.649797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.649683    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:12:54.649797 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.649704    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" event={"ID":"2fdcac7b-1332-4550-aab2-4a31878c564d","Type":"ContainerStarted","Data":"8ba32589af71f45f30008b433850e8fbc74a3aa813e9c2880272a51e48d5d3aa"}
Oct 31 04:12:54.689217 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.689146    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" podStartSLOduration=1.689130736 podStartE2EDuration="1.689130736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:54.686825212 +0000 UTC m=+3449.212044806" watchObservedRunningTime="2026-10-31 04:12:54.689130736 +0000 UTC m=+3449.214350318"
Oct 31 04:12:54.725844 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:54.725783    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" podStartSLOduration=1.72576796 podStartE2EDuration="1.72576796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:12:54.722182055 +0000 UTC m=+3449.247401643" watchObservedRunningTime="2026-10-31 04:12:54.72576796 +0000 UTC m=+3449.250987549"
Oct 31 04:12:55.535643 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:12:55.535596    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:13:01.057632 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:13:01.057568    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:13:05.661389 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:13:05.661307    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:13:05.661389 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:13:05.661385    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:13:31.057508 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:13:31.057462    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:13:42.266000 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:13:42.265965    5098 scope.go:122] "RemoveContainer" containerID="cb081fbaf3e3d20d3a7d682fcdbcb66b2e0baac1e56d56b9c11ee03a3bc78502"
Oct 31 04:14:01.058286 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:14:01.058248    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:14:29.641140 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.641100    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"]
Oct 31 04:14:29.645198 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.645169    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.650749 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.650717    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:14:29.650902 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.650789    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:14:29.654563 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.654535    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"]
Oct 31 04:14:29.662334 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.662299    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:14:29.662482 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.662425    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cq5w7\""
Oct 31 04:14:29.750052 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.750012    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqxz\" (UniqueName: \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.750243 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.750150    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.750294 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.750279    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.851101 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.851052    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.851309 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.851118    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqxz\" (UniqueName: \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.851309 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.851183    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.851922 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.851897    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.853548 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.853522    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.859062 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.859037    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqxz\" (UniqueName: \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:29.960929 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:29.960846    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:30.236596 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:30.236570    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"]
Oct 31 04:14:30.238601 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:14:30.238572    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c9a626_983d_4c0f_8f90_259ac8225e63.slice/crio-8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3 WatchSource:0}: Error finding container 8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3: Status 404 returned error can't find the container with id 8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3
Oct 31 04:14:31.058259 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:14:31.058217    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:14:31.236552 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:31.236518    5098 generic.go:356] "Generic (PLEG): container finished" podID="d1c9a626-983d-4c0f-8f90-259ac8225e63" containerID="2db26a56a706e5f96f70017ccda8c514ee59ccd33f5331fcf974f3c1b38971c8" exitCode=0
Oct 31 04:14:31.236773 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:31.236620    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh" event={"ID":"d1c9a626-983d-4c0f-8f90-259ac8225e63","Type":"ContainerDied","Data":"2db26a56a706e5f96f70017ccda8c514ee59ccd33f5331fcf974f3c1b38971c8"}
Oct 31 04:14:31.236773 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:31.236683    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh" event={"ID":"d1c9a626-983d-4c0f-8f90-259ac8225e63","Type":"ContainerStarted","Data":"8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3"}
Oct 31 04:14:32.393804 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.393778    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:32.578502 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.578463    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\") pod \"d1c9a626-983d-4c0f-8f90-259ac8225e63\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") "
Oct 31 04:14:32.578740 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.578519    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\" (UniqueName: \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\") pod \"d1c9a626-983d-4c0f-8f90-259ac8225e63\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") "
Oct 31 04:14:32.578740 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.578546    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\") pod \"d1c9a626-983d-4c0f-8f90-259ac8225e63\" (UID: \"d1c9a626-983d-4c0f-8f90-259ac8225e63\") "
Oct 31 04:14:32.578904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.578880    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca" pod "d1c9a626-983d-4c0f-8f90-259ac8225e63" (UID: "d1c9a626-983d-4c0f-8f90-259ac8225e63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:32.580846 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.580818    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz" pod "d1c9a626-983d-4c0f-8f90-259ac8225e63" (UID: "d1c9a626-983d-4c0f-8f90-259ac8225e63"). InnerVolumeSpecName "kube-api-access-flqxz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:32.580959 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.580920    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca" pod "d1c9a626-983d-4c0f-8f90-259ac8225e63" (UID: "d1c9a626-983d-4c0f-8f90-259ac8225e63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:14:32.679967 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.679932    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:32.679967 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.679964    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flqxz\" (UniqueName: \"kubernetes.io/projected/d1c9a626-983d-4c0f-8f90-259ac8225e63-kube-api-access-flqxz\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:32.679967 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:32.679975    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1c9a626-983d-4c0f-8f90-259ac8225e63-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:33.249833 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:33.249797    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"
Oct 31 04:14:33.250061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:33.249831    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh" event={"ID":"d1c9a626-983d-4c0f-8f90-259ac8225e63","Type":"ContainerDied","Data":"8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3"}
Oct 31 04:14:33.250061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:33.249882    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d338af6705d9345f5d0fa6e80b4afdcdbe13846e4930ee4e557fa0be4866ae3"
Oct 31 04:14:39.789662 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:39.789616    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:14:39.801715 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:39.801680    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:14:44.790602 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:44.790536    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" podUID="2fdcac7b-1332-4550-aab2-4a31878c564d" containerName="payload-processing" containerID="cri-o://d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21" gracePeriod=25
Oct 31 04:14:44.802257 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:44.802216    5098 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" podUID="9db32cbf-918a-4483-8601-322010cfd9cc" containerName="payload-pre-processing" containerID="cri-o://68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a" gracePeriod=25
Oct 31 04:14:45.110725 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.110699    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"]
Oct 31 04:14:45.112882 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.112853    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5e424f-4chlh"]
Oct 31 04:14:45.132503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.132477    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:14:45.136405 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.136370    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:14:45.208761 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.208727    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\" (UniqueName: \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\") pod \"2fdcac7b-1332-4550-aab2-4a31878c564d\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") "
Oct 31 04:14:45.208988 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.208793    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\") pod \"9db32cbf-918a-4483-8601-322010cfd9cc\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") "
Oct 31 04:14:45.208988 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.208851    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\" (UniqueName: \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\") pod \"9db32cbf-918a-4483-8601-322010cfd9cc\" (UID: \"9db32cbf-918a-4483-8601-322010cfd9cc\") "
Oct 31 04:14:45.208988 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.208904    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\") pod \"2fdcac7b-1332-4550-aab2-4a31878c564d\" (UID: \"2fdcac7b-1332-4550-aab2-4a31878c564d\") "
Oct 31 04:14:45.209536 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.209506    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume" pod "2fdcac7b-1332-4550-aab2-4a31878c564d" (UID: "2fdcac7b-1332-4550-aab2-4a31878c564d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:45.209699 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.209573    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume" pod "9db32cbf-918a-4483-8601-322010cfd9cc" (UID: "9db32cbf-918a-4483-8601-322010cfd9cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:14:45.211761 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.211714    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk" pod "2fdcac7b-1332-4550-aab2-4a31878c564d" (UID: "2fdcac7b-1332-4550-aab2-4a31878c564d"). InnerVolumeSpecName "kube-api-access-6k5nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:45.211907 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.211847    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv" pod "9db32cbf-918a-4483-8601-322010cfd9cc" (UID: "9db32cbf-918a-4483-8601-322010cfd9cc"). InnerVolumeSpecName "kube-api-access-swsfv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:14:45.310847 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.310807    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k5nk\" (UniqueName: \"kubernetes.io/projected/2fdcac7b-1332-4550-aab2-4a31878c564d-kube-api-access-6k5nk\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:45.310847 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.310844    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9db32cbf-918a-4483-8601-322010cfd9cc-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:45.311093 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.310861    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsfv\" (UniqueName: \"kubernetes.io/projected/9db32cbf-918a-4483-8601-322010cfd9cc-kube-api-access-swsfv\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:45.311093 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.310879    5098 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2fdcac7b-1332-4550-aab2-4a31878c564d-plugins-config-volume\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:14:45.321130 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.321095    5098 generic.go:356] "Generic (PLEG): container finished" podID="2fdcac7b-1332-4550-aab2-4a31878c564d" containerID="d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21" exitCode=0
Oct 31 04:14:45.321310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.321140    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"
Oct 31 04:14:45.321310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.321182    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" event={"ID":"2fdcac7b-1332-4550-aab2-4a31878c564d","Type":"ContainerDied","Data":"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"}
Oct 31 04:14:45.321310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.321248    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64" event={"ID":"2fdcac7b-1332-4550-aab2-4a31878c564d","Type":"ContainerDied","Data":"8ba32589af71f45f30008b433850e8fbc74a3aa813e9c2880272a51e48d5d3aa"}
Oct 31 04:14:45.321310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.321278    5098 scope.go:122] "RemoveContainer" containerID="d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"
Oct 31 04:14:45.322909 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.322889    5098 generic.go:356] "Generic (PLEG): container finished" podID="9db32cbf-918a-4483-8601-322010cfd9cc" containerID="68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a" exitCode=0
Oct 31 04:14:45.323022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.322919    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" event={"ID":"9db32cbf-918a-4483-8601-322010cfd9cc","Type":"ContainerDied","Data":"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"}
Oct 31 04:14:45.323022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.322955    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt" event={"ID":"9db32cbf-918a-4483-8601-322010cfd9cc","Type":"ContainerDied","Data":"d7798cfc36449647afa5d59b0d2891c476b6000d9b450443d6b12d1b833cac71"}
Oct 31 04:14:45.323022 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.323010    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"
Oct 31 04:14:45.334195 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.334176    5098 scope.go:122] "RemoveContainer" containerID="d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"
Oct 31 04:14:45.334484 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:14:45.334461    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21\": container with ID starting with d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21 not found: ID does not exist" containerID="d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"
Oct 31 04:14:45.334583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.334497    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21"} err="failed to get container status \"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21\": rpc error: code = NotFound desc = could not find container \"d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21\": container with ID starting with d716fdb9a45091da6fb15ce55acc063f51809e2f444c35eac649f9e3d97cfd21 not found: ID does not exist"
Oct 31 04:14:45.334583 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.334520    5098 scope.go:122] "RemoveContainer" containerID="68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"
Oct 31 04:14:45.344443 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.344422    5098 scope.go:122] "RemoveContainer" containerID="68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"
Oct 31 04:14:45.344780 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:14:45.344756    5098 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a\": container with ID starting with 68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a not found: ID does not exist" containerID="68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"
Oct 31 04:14:45.344866 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.344790    5098 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a"} err="failed to get container status \"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a\": rpc error: code = NotFound desc = could not find container \"68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a\": container with ID starting with 68ebee8c3236de0d792a14443831841ea428154be0b6af724838ab6e5816914a not found: ID does not exist"
Oct 31 04:14:45.351213 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.351147    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:14:45.353430 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.353408    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5e424f-7c759dd8d7-2zv64"]
Oct 31 04:14:45.367493 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.367462    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:14:45.373649 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.373617    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5e424f-6dcb7d48b7-6wfbt"]
Oct 31 04:14:45.612756 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.612663    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdcac7b-1332-4550-aab2-4a31878c564d" path="/var/lib/kubelet/pods/2fdcac7b-1332-4550-aab2-4a31878c564d/volumes"
Oct 31 04:14:45.613233 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.613211    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db32cbf-918a-4483-8601-322010cfd9cc" path="/var/lib/kubelet/pods/9db32cbf-918a-4483-8601-322010cfd9cc/volumes"
Oct 31 04:14:45.613795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:45.613765    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1c9a626-983d-4c0f-8f90-259ac8225e63" path="/var/lib/kubelet/pods/d1c9a626-983d-4c0f-8f90-259ac8225e63/volumes"
Oct 31 04:14:46.442118 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:46.442083    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:14:53.647381 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:53.647343    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:14:56.141193 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:56.141145    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-6vdn7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:14:56.141651 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:56.141209    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-6vdn7" podUID="9a8b9637-0546-43b8-b5b4-c67bce1b0345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:14:56.155640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:56.154710    5098 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-769df7846c-tnsvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 04:14:56.155640 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:56.154793    5098 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-769df7846c-tnsvh" podUID="a50c6120-b67e-4ca2-a369-5ae4b01d5fdb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 04:14:58.837714 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:14:58.837671    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:15:00.144991 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.144943    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:00.145664 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145640    5098 cpu_manager.go:414] "containerMap: removing container" podUID="9db32cbf-918a-4483-8601-322010cfd9cc" containerName="payload-pre-processing"
Oct 31 04:15:00.145664 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145666    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db32cbf-918a-4483-8601-322010cfd9cc" containerName="payload-pre-processing"
Oct 31 04:15:00.145834 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145698    5098 cpu_manager.go:414] "containerMap: removing container" podUID="2fdcac7b-1332-4550-aab2-4a31878c564d" containerName="payload-processing"
Oct 31 04:15:00.145834 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145707    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdcac7b-1332-4550-aab2-4a31878c564d" containerName="payload-processing"
Oct 31 04:15:00.145834 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145749    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d1c9a626-983d-4c0f-8f90-259ac8225e63" containerName="revoke-keys"
Oct 31 04:15:00.145834 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145758    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1c9a626-983d-4c0f-8f90-259ac8225e63" containerName="revoke-keys"
Oct 31 04:15:00.146069 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145889    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db32cbf-918a-4483-8601-322010cfd9cc" containerName="payload-pre-processing"
Oct 31 04:15:00.146069 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145905    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdcac7b-1332-4550-aab2-4a31878c564d" containerName="payload-processing"
Oct 31 04:15:00.146069 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.145920    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1c9a626-983d-4c0f-8f90-259ac8225e63" containerName="revoke-keys"
Oct 31 04:15:00.149830 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.149798    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.156474 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.156321    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cq5w7\""
Oct 31 04:15:00.156864 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.156423    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:15:00.156864 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.156810    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:15:00.164808 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.164777    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:00.215001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.214295    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrhb\" (UniqueName: \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.215001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.214466    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.336632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.333793    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.336632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.333880    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrhb\" (UniqueName: \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.336632 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.335094    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.343226 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.343190    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrhb\" (UniqueName: \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\") pod \"maas-api-key-cleanup-29890335-hk94k\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:00.576875 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:00.576826    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:01.047816 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:01.041300    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:01.265832 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:15:01.265746    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:15:01.451021 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:01.450925    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerStarted","Data":"a1bf073bc05eeff7604b56db7741ed6dfd60cc47a470129890378602d22dfa8d"}
Oct 31 04:15:02.459368 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:02.459323    5098 generic.go:356] "Generic (PLEG): container finished" podID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerID="652b2b9bf726b715b2de70904f1dc98c025815fdbe6b8732b6b22047533c5346" exitCode=60
Oct 31 04:15:02.459887 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:02.459406    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerDied","Data":"652b2b9bf726b715b2de70904f1dc98c025815fdbe6b8732b6b22047533c5346"}
Oct 31 04:15:02.459887 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:02.459675    5098 scope.go:122] "RemoveContainer" containerID="652b2b9bf726b715b2de70904f1dc98c025815fdbe6b8732b6b22047533c5346"
Oct 31 04:15:03.467055 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:03.467019    5098 generic.go:356] "Generic (PLEG): container finished" podID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerID="180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766" exitCode=60
Oct 31 04:15:03.467541 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:03.467072    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerDied","Data":"180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766"}
Oct 31 04:15:03.467541 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:03.467135    5098 scope.go:122] "RemoveContainer" containerID="652b2b9bf726b715b2de70904f1dc98c025815fdbe6b8732b6b22047533c5346"
Oct 31 04:15:03.467541 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:03.467432    5098 scope.go:122] "RemoveContainer" containerID="180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766"
Oct 31 04:15:03.467807 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:15:03.467786    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-hk94k_odh-ai-gateway-infra(fa486c3f-e495-47d7-84ca-bae6bdd989a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4"
Oct 31 04:15:04.474922 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:04.474881    5098 scope.go:122] "RemoveContainer" containerID="180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766"
Oct 31 04:15:04.475342 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:15:04.475144    5098 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890335-hk94k_odh-ai-gateway-infra(fa486c3f-e495-47d7-84ca-bae6bdd989a4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4"
Oct 31 04:15:14.137858 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:14.137818    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:15:16.604346 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:16.604299    5098 scope.go:122] "RemoveContainer" containerID="180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766"
Oct 31 04:15:17.572299 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:17.572242    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerStarted","Data":"f4318cb5baec827b0e3f2a6d746ef6c4a6e02180c552464fa288e1272aa7713d"}
Oct 31 04:15:17.603643 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:17.593517    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" podStartSLOduration=17.593497391 podStartE2EDuration="17.593497391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:17.589991066 +0000 UTC m=+3592.115210657" watchObservedRunningTime="2026-10-31 04:15:17.593497391 +0000 UTC m=+3592.118716979"
Oct 31 04:15:18.616914 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:18.616867    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:18.623702 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:18.623663    5098 generic.go:356] "Generic (PLEG): container finished" podID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerID="f4318cb5baec827b0e3f2a6d746ef6c4a6e02180c552464fa288e1272aa7713d" exitCode=60
Oct 31 04:15:18.623922 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:18.623798    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerDied","Data":"f4318cb5baec827b0e3f2a6d746ef6c4a6e02180c552464fa288e1272aa7713d"}
Oct 31 04:15:18.623922 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:18.623854    5098 scope.go:122] "RemoveContainer" containerID="180bbdff8415c59c40e1cc91adb4ef5ba9191f1061edca36209daa313567c766"
Oct 31 04:15:18.946739 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:18.946707    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:19.131565 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.131521    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\" (UniqueName: \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\") pod \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") "
Oct 31 04:15:19.131821 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.131641    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\") pod \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\" (UID: \"fa486c3f-e495-47d7-84ca-bae6bdd989a4\") "
Oct 31 04:15:19.132013 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.131987    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca" pod "fa486c3f-e495-47d7-84ca-bae6bdd989a4" (UID: "fa486c3f-e495-47d7-84ca-bae6bdd989a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:19.133771 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.133740    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb" pod "fa486c3f-e495-47d7-84ca-bae6bdd989a4" (UID: "fa486c3f-e495-47d7-84ca-bae6bdd989a4"). InnerVolumeSpecName "kube-api-access-sdrhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:19.232459 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.232376    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrhb\" (UniqueName: \"kubernetes.io/projected/fa486c3f-e495-47d7-84ca-bae6bdd989a4-kube-api-access-sdrhb\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.232459 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.232407    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa486c3f-e495-47d7-84ca-bae6bdd989a4-maas-api-service-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:19.264528 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.264490    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"]
Oct 31 04:15:19.265005 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.264988    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265060 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265007    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265060 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265021    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265060 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265026    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265129    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265137    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.265161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.265148    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:19.268407 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.268385    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.272483 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.272423    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:15:19.277095 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.276931    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"]
Oct 31 04:15:19.434599 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.434551    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.434871 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.434727    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.434871 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.434803    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdwhl\" (UniqueName: \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.536299 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.536265    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.536490 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.536315    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdwhl\" (UniqueName: \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.536490 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.536366    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.537001 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.536976    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.538878 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.538855    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.545657 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.545629    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdwhl\" (UniqueName: \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\") pod \"maas-api-revoke-sub-0165d7d2-dtm9k\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.585638 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.585584    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:19.632494 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.632452    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k" event={"ID":"fa486c3f-e495-47d7-84ca-bae6bdd989a4","Type":"ContainerDied","Data":"a1bf073bc05eeff7604b56db7741ed6dfd60cc47a470129890378602d22dfa8d"}
Oct 31 04:15:19.633017 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.632518    5098 scope.go:122] "RemoveContainer" containerID="f4318cb5baec827b0e3f2a6d746ef6c4a6e02180c552464fa288e1272aa7713d"
Oct 31 04:15:19.633017 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.632468    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"
Oct 31 04:15:19.661795 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.661733    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:19.664263 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.664226    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890335-hk94k"]
Oct 31 04:15:19.866863 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:19.866837    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"]
Oct 31 04:15:19.868954 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:15:19.868921    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8f6074f_3be7_4251_9eb9_550de794f4a8.slice/crio-44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24 WatchSource:0}: Error finding container 44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24: Status 404 returned error can't find the container with id 44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24
Oct 31 04:15:20.641579 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:20.641543    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k" event={"ID":"d8f6074f-3be7-4251-9eb9-550de794f4a8","Type":"ContainerStarted","Data":"83ca70a75ecaddbc228f54a3142cfff3ce52a7a935df8dcb6dea87c346d13aab"}
Oct 31 04:15:20.642082 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:20.641647    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k" event={"ID":"d8f6074f-3be7-4251-9eb9-550de794f4a8","Type":"ContainerStarted","Data":"44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24"}
Oct 31 04:15:20.658967 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:20.658914    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k" podStartSLOduration=1.658899336 podStartE2EDuration="1.658899336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 04:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:15:20.656939848 +0000 UTC m=+3595.182159438" watchObservedRunningTime="2026-10-31 04:15:20.658899336 +0000 UTC m=+3595.184118921"
Oct 31 04:15:21.021508 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.021406    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"]
Oct 31 04:15:21.023864 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.023834    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77ddec5e-5mj56"]
Oct 31 04:15:21.611203 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.611156    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="638decdc-a558-4f3d-829b-42b96c2aa863" path="/var/lib/kubelet/pods/638decdc-a558-4f3d-829b-42b96c2aa863/volumes"
Oct 31 04:15:21.611663 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.611646    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" path="/var/lib/kubelet/pods/fa486c3f-e495-47d7-84ca-bae6bdd989a4/volumes"
Oct 31 04:15:21.648536 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.648504    5098 generic.go:356] "Generic (PLEG): container finished" podID="d8f6074f-3be7-4251-9eb9-550de794f4a8" containerID="83ca70a75ecaddbc228f54a3142cfff3ce52a7a935df8dcb6dea87c346d13aab" exitCode=0
Oct 31 04:15:21.648965 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:21.648561    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k" event={"ID":"d8f6074f-3be7-4251-9eb9-550de794f4a8","Type":"ContainerDied","Data":"83ca70a75ecaddbc228f54a3142cfff3ce52a7a935df8dcb6dea87c346d13aab"}
Oct 31 04:15:22.797801 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.797774    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:22.971861 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.971777    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\" (UniqueName: \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\") pod \"d8f6074f-3be7-4251-9eb9-550de794f4a8\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") "
Oct 31 04:15:22.971861 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.971822    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\") pod \"d8f6074f-3be7-4251-9eb9-550de794f4a8\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") "
Oct 31 04:15:22.972068 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.971928    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\") pod \"d8f6074f-3be7-4251-9eb9-550de794f4a8\" (UID: \"d8f6074f-3be7-4251-9eb9-550de794f4a8\") "
Oct 31 04:15:22.972293 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.972260    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca" pod "d8f6074f-3be7-4251-9eb9-550de794f4a8" (UID: "d8f6074f-3be7-4251-9eb9-550de794f4a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:22.974114 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.974082    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl" pod "d8f6074f-3be7-4251-9eb9-550de794f4a8" (UID: "d8f6074f-3be7-4251-9eb9-550de794f4a8"). InnerVolumeSpecName "kube-api-access-fdwhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:22.974225 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:22.974116    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca" pod "d8f6074f-3be7-4251-9eb9-550de794f4a8" (UID: "d8f6074f-3be7-4251-9eb9-550de794f4a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:15:23.072891 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.072850    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:23.072891 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.072886    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdwhl\" (UniqueName: \"kubernetes.io/projected/d8f6074f-3be7-4251-9eb9-550de794f4a8-kube-api-access-fdwhl\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:23.072891 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.072899    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8f6074f-3be7-4251-9eb9-550de794f4a8-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:23.662432 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.662401    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k"
Oct 31 04:15:23.662601 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.662402    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-dtm9k" event={"ID":"d8f6074f-3be7-4251-9eb9-550de794f4a8","Type":"ContainerDied","Data":"44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24"}
Oct 31 04:15:23.662601 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:23.662524    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44db75a57f164bcc69f19a9a0328485308511bcc32961a63ffdc27e9ba72da24"
Oct 31 04:15:29.963984 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.963941    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"]
Oct 31 04:15:29.964581 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.964553    5098 cpu_manager.go:414] "containerMap: removing container" podUID="d8f6074f-3be7-4251-9eb9-550de794f4a8" containerName="revoke-keys"
Oct 31 04:15:29.964581 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.964572    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8f6074f-3be7-4251-9eb9-550de794f4a8" containerName="revoke-keys"
Oct 31 04:15:29.964815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.964656    5098 cpu_manager.go:414] "containerMap: removing container" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:29.964815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.964665    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa486c3f-e495-47d7-84ca-bae6bdd989a4" containerName="cleanup"
Oct 31 04:15:29.964815 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.964765    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8f6074f-3be7-4251-9eb9-550de794f4a8" containerName="revoke-keys"
Oct 31 04:15:29.968298 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.968272    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:29.976741 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.976704    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 04:15:29.976944 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.976797    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 31 04:15:29.976944 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.976891    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-cq5w7\""
Oct 31 04:15:29.977514 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.977491    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 04:15:29.977931 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:29.977908    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"]
Oct 31 04:15:30.043670 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.043633    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkc72\" (UniqueName: \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.043855 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.043692    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.043855 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.043842    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.145169 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.145126    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkc72\" (UniqueName: \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.145385 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.145178    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.145385 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.145263    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.145956 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.145927    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.147891 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.147867    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.153536 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.153515    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkc72\" (UniqueName: \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\") pod \"maas-api-revoke-sub-6b19338f-j2c5g\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.297667 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.297618    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:30.594709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.594680    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"]
Oct 31 04:15:30.598173 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:15:30.598111    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6474c5a_1d1f_4b6c_8dc0_ee743c3190a4.slice/crio-1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23 WatchSource:0}: Error finding container 1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23: Status 404 returned error can't find the container with id 1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23
Oct 31 04:15:30.705329 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:30.705288    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g" event={"ID":"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4","Type":"ContainerStarted","Data":"1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23"}
Oct 31 04:15:31.057774 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:15:31.057733    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:15:31.713386 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:31.713352    5098 generic.go:356] "Generic (PLEG): container finished" podID="c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" containerID="020f2a33c284bedebf7f356cd334b9f939b619cde54cec8c3272107967586c44" exitCode=0
Oct 31 04:15:31.713585 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:31.713433    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g" event={"ID":"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4","Type":"ContainerDied","Data":"020f2a33c284bedebf7f356cd334b9f939b619cde54cec8c3272107967586c44"}
Oct 31 04:15:32.861043 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.861018    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:32.973462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.973356    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\") pod \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") "
Oct 31 04:15:32.973462 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.973399    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\") pod \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") "
Oct 31 04:15:32.973774 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.973484    5098 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\" (UniqueName: \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\") pod \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\" (UID: \"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4\") "
Oct 31 04:15:32.973841 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.973785    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca" pod "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" (UID: "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 04:15:32.973900 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.973883    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-openshift-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:32.975784 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.975760    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca" pod "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" (UID: "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 04:15:32.975931 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:32.975891    5098 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72" pod "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" (UID: "c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4"). InnerVolumeSpecName "kube-api-access-fkc72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 04:15:33.075123 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.075082    5098 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-maas-api-xks-ca\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:33.075123 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.075114    5098 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkc72\" (UniqueName: \"kubernetes.io/projected/c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4-kube-api-access-fkc72\") on node \"ip-10-0-129-132.us-east-2.compute.internal\" DevicePath \"\""
Oct 31 04:15:33.120278 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.120249    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:15:33.120997 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.120967    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:15:33.725921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.725890    5098 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g"
Oct 31 04:15:33.726111 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.725890    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-j2c5g" event={"ID":"c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4","Type":"ContainerDied","Data":"1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23"}
Oct 31 04:15:33.726111 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:33.726013    5098 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1872c7538ff7051484e30f2fa7b088f7920c5442d24abcac437249a87f92fa23"
Oct 31 04:15:40.839470 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:40.839433    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-hbwwx"]
Oct 31 04:15:51.202183 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:51.202145    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6fd9f75659-4s8nf_ceb67824-a366-4e5b-b4fa-4d8a48e898eb/manager/0.log"
Oct 31 04:15:54.238004 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:54.237969    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dtm9k_d8f6074f-3be7-4251-9eb9-550de794f4a8/revoke-keys/0.log"
Oct 31 04:15:54.410917 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:54.410886    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-rl8hs_8028d51f-aab9-40b7-836a-18bbd7c4c8d1/revoke-keys/0.log"
Oct 31 04:15:54.585490 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:54.585449    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-kgcr5_049a3de7-75ee-4fb1-a321-945677422802/revoke-keys/0.log"
Oct 31 04:15:54.928385 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:54.928291    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-j2c5g_c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4/revoke-keys/0.log"
Oct 31 04:15:56.932136 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:56.932098    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ld4pq_f07ce24f-e523-48d4-9458-bab5f9b7f2af/manager/0.log"
Oct 31 04:15:57.091519 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:57.091480    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-grs9n_1bd96048-f68d-46b4-be69-75ebc2987909/manager/0.log"
Oct 31 04:15:57.465154 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:57.465109    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2h7s4_6268bbb3-55f4-4c7f-9e77-319f00f7a5a0/manager/0.log"
Oct 31 04:15:57.666327 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:57.666294    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hbwwx_bd289733-c246-4cb5-a87f-eeb4a8d1ca4f/limitador/0.log"
Oct 31 04:15:58.787017 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:58.786982    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5czs5l_6832d278-dba5-4b0d-af12-726ea83e1764/istio-proxy/0.log"
Oct 31 04:15:59.694365 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:59.694333    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-6vdn7_9a8b9637-0546-43b8-b5b4-c67bce1b0345/istio-proxy/0.log"
Oct 31 04:15:59.870351 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:15:59.870310    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-b8m6d_aee37c72-f7bb-459c-9f06-df7401cc1daf/istio-proxy/0.log"
Oct 31 04:16:01.057093 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:16:01.057054    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:16:01.081526 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:01.081493    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tnsvh_a50c6120-b67e-4ca2-a369-5ae4b01d5fdb/istio-proxy/0.log"
Oct 31 04:16:01.257902 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:01.257872    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg_3afa466d-1d28-4af9-a622-b3be1061dd07/istio-proxy/0.log"
Oct 31 04:16:01.757361 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:01.757328    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5497575d9c-lm8wr_b91cefe7-2e73-4f49-8d39-1187617dc044/router/0.log"
Oct 31 04:16:02.680469 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:02.680427    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k66s8_17595e19-e8f1-47af-9b70-0b75e8559dde/main/0.log"
Oct 31 04:16:02.850977 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:02.850946    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8jbq7_9c5fe135-1668-41e1-a2d4-42aa4234e43e/main/0.log"
Oct 31 04:16:03.179165 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:03.179123    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl_5899e70c-2b85-4866-aa96-3a755adfe0f8/main/0.log"
Oct 31 04:16:03.657300 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:03.657270    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln_511eac43-6d5d-4d2f-9c1e-ca4c515c9fea/main/0.log"
Oct 31 04:16:13.193516 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:13.193427    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-6fd9f75659-4s8nf_ceb67824-a366-4e5b-b4fa-4d8a48e898eb/manager/0.log"
Oct 31 04:16:15.496870 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:15.496835    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-dtm9k_d8f6074f-3be7-4251-9eb9-550de794f4a8/revoke-keys/0.log"
Oct 31 04:16:15.684240 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:15.684207    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-rl8hs_8028d51f-aab9-40b7-836a-18bbd7c4c8d1/revoke-keys/0.log"
Oct 31 04:16:15.858808 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:15.858773    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-kgcr5_049a3de7-75ee-4fb1-a321-945677422802/revoke-keys/0.log"
Oct 31 04:16:16.187854 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:16.187772    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-j2c5g_c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4/revoke-keys/0.log"
Oct 31 04:16:17.881812 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:17.881762    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ld4pq_f07ce24f-e523-48d4-9458-bab5f9b7f2af/manager/0.log"
Oct 31 04:16:18.063568 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:18.063537    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-grs9n_1bd96048-f68d-46b4-be69-75ebc2987909/manager/0.log"
Oct 31 04:16:18.427243 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:18.427202    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2h7s4_6268bbb3-55f4-4c7f-9e77-319f00f7a5a0/manager/0.log"
Oct 31 04:16:18.621454 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:18.621422    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hbwwx_bd289733-c246-4cb5-a87f-eeb4a8d1ca4f/limitador/0.log"
Oct 31 04:16:19.350984 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:19.350951    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5czs5l_6832d278-dba5-4b0d-af12-726ea83e1764/istio-proxy/0.log"
Oct 31 04:16:20.266095 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:20.266047    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-6vdn7_9a8b9637-0546-43b8-b5b4-c67bce1b0345/istio-proxy/0.log"
Oct 31 04:16:20.457578 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:20.457543    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-b8m6d_aee37c72-f7bb-459c-9f06-df7401cc1daf/istio-proxy/0.log"
Oct 31 04:16:21.742677 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:21.742641    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tnsvh_a50c6120-b67e-4ca2-a369-5ae4b01d5fdb/istio-proxy/0.log"
Oct 31 04:16:21.914227 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:21.914191    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg_3afa466d-1d28-4af9-a622-b3be1061dd07/istio-proxy/0.log"
Oct 31 04:16:22.430684 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:22.430651    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5497575d9c-lm8wr_b91cefe7-2e73-4f49-8d39-1187617dc044/router/0.log"
Oct 31 04:16:23.105830 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:23.105798    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-k66s8_17595e19-e8f1-47af-9b70-0b75e8559dde/main/0.log"
Oct 31 04:16:23.290750 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:23.290718    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-8jbq7_9c5fe135-1668-41e1-a2d4-42aa4234e43e/main/0.log"
Oct 31 04:16:23.628876 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:23.628826    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8h5zbl_5899e70c-2b85-4866-aa96-3a755adfe0f8/main/0.log"
Oct 31 04:16:24.156664 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:24.156628    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-lwvln_511eac43-6d5d-4d2f-9c1e-ca4c515c9fea/main/0.log"
Oct 31 04:16:28.022012 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:28.021968    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"]
Oct 31 04:16:28.024213 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:28.024185    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-kgcr5"]
Oct 31 04:16:29.610791 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:29.610755    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="049a3de7-75ee-4fb1-a321-945677422802" path="/var/lib/kubelet/pods/049a3de7-75ee-4fb1-a321-945677422802/volumes"
Oct 31 04:16:31.056895 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:16:31.056855    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:16:41.020938 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:41.020888    5098 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"]
Oct 31 04:16:41.023153 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:41.023124    5098 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-rl8hs"]
Oct 31 04:16:41.609777 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:41.609742    5098 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8028d51f-aab9-40b7-836a-18bbd7c4c8d1" path="/var/lib/kubelet/pods/8028d51f-aab9-40b7-836a-18bbd7c4c8d1/volumes"
Oct 31 04:16:42.359376 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:42.359341    5098 scope.go:122] "RemoveContainer" containerID="db115d2fa758939b1fe3a8aee19ad601fa339fce0ea0ac19bac2f036d39a5f7a"
Oct 31 04:16:42.369478 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:16:42.369457    5098 scope.go:122] "RemoveContainer" containerID="25bd21e1474e335e3e2e7a91a915e59da928fd2a1514e2e7d76bff58bff95802"
Oct 31 04:17:01.060436 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:17:01.060386    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:17:31.057326 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:17:31.057281    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:17:42.397161 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:42.397076    5098 scope.go:122] "RemoveContainer" containerID="704fd4ba0ff9bd8295b71795dedab4fd98d34f874b2872946c63a5623d7a6e39"
Oct 31 04:17:42.407751 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:42.407723    5098 scope.go:122] "RemoveContainer" containerID="02713c10e746c866bab5e03daea00302b245a3e27d75a8dbe743610ff36c1f74"
Oct 31 04:17:57.394010 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.393977    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bjxm9_c81c0c55-4184-4b1b-a431-268e608e3b60/global-pull-secret-syncer/0.log"
Oct 31 04:17:57.468114 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.468082    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-5f46r_0736e363-b4be-4a6f-8fe5-e9a1aeea0222/connection-checker/0.log"
Oct 31 04:17:57.492727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.492695    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-dkwqg_e077c72b-608e-43e2-b55f-db29e830b507/connection-checker/0.log"
Oct 31 04:17:57.515709 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.515673    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-jmx67_c89742f3-947b-45c2-a14b-491054ba970f/connection-checker/0.log"
Oct 31 04:17:57.637790 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.637748    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6csjw_6d23fdfb-3765-488e-bad2-d80a7fe3a162/konnectivity-agent/0.log"
Oct 31 04:17:57.724497 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:17:57.724408    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-132.us-east-2.compute.internal_776064ad5d3782b191602f1d077b5a65/haproxy/0.log"
Oct 31 04:18:01.056873 ip-10-0-129-132 kubenswrapper[5098]: E1031 04:18:01.056826    5098 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4crls" containerName="extractor"
Oct 31 04:18:04.981529 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:04.981492    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-ld4pq_f07ce24f-e523-48d4-9458-bab5f9b7f2af/manager/0.log"
Oct 31 04:18:05.007832 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:05.007791    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-grs9n_1bd96048-f68d-46b4-be69-75ebc2987909/manager/0.log"
Oct 31 04:18:05.120279 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:05.120247    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-2h7s4_6268bbb3-55f4-4c7f-9e77-319f00f7a5a0/manager/0.log"
Oct 31 04:18:05.150222 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:05.150179    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-hbwwx_bd289733-c246-4cb5-a87f-eeb4a8d1ca4f/limitador/0.log"
Oct 31 04:18:06.734770 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.734731    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-2cg4n_378f839e-767e-421d-b8c7-65369e94a937/cluster-monitoring-operator/0.log"
Oct 31 04:18:06.898399 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.898365    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lkzg_a0ac836a-6fac-414b-886c-99fee2b253d2/node-exporter/0.log"
Oct 31 04:18:06.920273 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.920240    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lkzg_a0ac836a-6fac-414b-886c-99fee2b253d2/kube-rbac-proxy/0.log"
Oct 31 04:18:06.942806 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.942781    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9lkzg_a0ac836a-6fac-414b-886c-99fee2b253d2/init-textfile/0.log"
Oct 31 04:18:06.996259 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.996177    5098 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"]
Oct 31 04:18:06.996678 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.996664    5098 cpu_manager.go:414] "containerMap: removing container" podUID="c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" containerName="revoke-keys"
Oct 31 04:18:06.996735 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.996680    5098 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" containerName="revoke-keys"
Oct 31 04:18:06.996779 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.996760    5098 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6474c5a-1d1f-4b6c-8dc0-ee743c3190a4" containerName="revoke-keys"
Oct 31 04:18:07.000006 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:06.999990    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.005080 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.005057    5098 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6qbd4\"/\"default-dockercfg-f7krs\""
Oct 31 04:18:07.008463 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.008438    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"kube-root-ca.crt\""
Oct 31 04:18:07.008644 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.008459    5098 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6qbd4\"/\"openshift-service-ca.crt\""
Oct 31 04:18:07.012398 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.012373    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"]
Oct 31 04:18:07.046291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.046234    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-sys\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.046291 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.046292    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-proc\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.046600 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.046423    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lqpw\" (UniqueName: \"kubernetes.io/projected/9bc79ff6-9056-489e-804b-c1560c4b70e7-kube-api-access-7lqpw\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.046600 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.046476    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-podres\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.046600 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.046562    5098 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-lib-modules\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147687 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147643    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-lib-modules\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147769    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-sys\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147794    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-proc\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147812    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-lib-modules\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147873    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lqpw\" (UniqueName: \"kubernetes.io/projected/9bc79ff6-9056-489e-804b-c1560c4b70e7-kube-api-access-7lqpw\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.147884 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147881    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-proc\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.148071 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147896    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-sys\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.148071 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.147935    5098 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-podres\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.148071 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.148031    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/9bc79ff6-9056-489e-804b-c1560c4b70e7-podres\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.156532 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.156507    5098 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lqpw\" (UniqueName: \"kubernetes.io/projected/9bc79ff6-9056-489e-804b-c1560c4b70e7-kube-api-access-7lqpw\") pod \"perf-node-gather-daemonset-jq7pf\" (UID: \"9bc79ff6-9056-489e-804b-c1560c4b70e7\") " pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.314649 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.314584    5098 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:07.470225 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.470191    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-6jnt8_5d7ff043-ea26-4c96-874d-c2d6ab18b6a9/prometheus-operator-admission-webhook/0.log"
Oct 31 04:18:07.589478 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.589444    5098 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"]
Oct 31 04:18:07.591222 ip-10-0-129-132 kubenswrapper[5098]: W1031 04:18:07.591193    5098 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9bc79ff6_9056_489e_804b_c1560c4b70e7.slice/crio-f9af2f5086bfe3ed9145f8d7de307ad51df55aec329260b1a2406b02ea0c0f82 WatchSource:0}: Error finding container f9af2f5086bfe3ed9145f8d7de307ad51df55aec329260b1a2406b02ea0c0f82: Status 404 returned error can't find the container with id f9af2f5086bfe3ed9145f8d7de307ad51df55aec329260b1a2406b02ea0c0f82
Oct 31 04:18:07.641551 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:07.641520    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf" event={"ID":"9bc79ff6-9056-489e-804b-c1560c4b70e7","Type":"ContainerStarted","Data":"f9af2f5086bfe3ed9145f8d7de307ad51df55aec329260b1a2406b02ea0c0f82"}
Oct 31 04:18:08.649049 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:08.649006    5098 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf" event={"ID":"9bc79ff6-9056-489e-804b-c1560c4b70e7","Type":"ContainerStarted","Data":"ffe0801a2af78f018f64435fe323cd43608a7b2378203857e50159cd19a2c610"}
Oct 31 04:18:08.649475 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:08.649202    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:08.666878 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:08.666822    5098 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf" podStartSLOduration=2.666806917 podStartE2EDuration="2.666806917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 04:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 04:18:08.66488787 +0000 UTC m=+3763.190107460" watchObservedRunningTime="2026-10-31 04:18:08.666806917 +0000 UTC m=+3763.192026506"
Oct 31 04:18:08.795580 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:08.795540    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-4927g_9a4b3f3c-5b83-481b-9038-aad4508357bf/networking-console-plugin/0.log"
Oct 31 04:18:09.418503 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:09.418472    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/2.log"
Oct 31 04:18:09.426415 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:09.426387    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-nq5jp_1b4b412a-ef91-4c3e-89c3-9287445d404c/console-operator/3.log"
Oct 31 04:18:09.916566 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:09.916519    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-mv884_54e44444-9207-4253-aaa5-77b849d81581/download-server/0.log"
Oct 31 04:18:10.490806 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:10.490774    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-ktx5k_97e7fbff-1276-4704-9580-c2a43dc17a97/volume-data-source-validator/0.log"
Oct 31 04:18:11.713921 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:11.713887    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbdjh_4d7fc633-6f9f-4d5a-adfc-394293b6e4af/dns/0.log"
Oct 31 04:18:11.733145 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:11.733111    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-lbdjh_4d7fc633-6f9f-4d5a-adfc-394293b6e4af/kube-rbac-proxy/0.log"
Oct 31 04:18:11.833369 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:11.833343    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nnxgp_b1d993de-0f1b-4b88-a998-344eec8c23ac/dns-node-resolver/0.log"
Oct 31 04:18:12.410372 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:12.410340    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-56fb5f95b9-s9pcp_5f6f81f6-dd4e-460e-9f61-b09e7300702e/registry/0.log"
Oct 31 04:18:12.463727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:12.463680    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wtqp4_79a249fe-49bc-4f12-b1f5-c2e4b3ecc990/node-ca/0.log"
Oct 31 04:18:13.452582 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:13.452548    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5czs5l_6832d278-dba5-4b0d-af12-726ea83e1764/istio-proxy/0.log"
Oct 31 04:18:14.664904 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:14.664871    5098 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-6qbd4/perf-node-gather-daemonset-jq7pf"
Oct 31 04:18:17.118289 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.118254    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-6vdn7_9a8b9637-0546-43b8-b5b4-c67bce1b0345/istio-proxy/0.log"
Oct 31 04:18:17.149737 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.149703    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-b8m6d_aee37c72-f7bb-459c-9f06-df7401cc1daf/istio-proxy/0.log"
Oct 31 04:18:17.369631 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.369531    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tnsvh_a50c6120-b67e-4ca2-a369-5ae4b01d5fdb/istio-proxy/0.log"
Oct 31 04:18:17.395597 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.395563    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg_3afa466d-1d28-4af9-a622-b3be1061dd07/istio-proxy/0.log"
Oct 31 04:18:17.482310 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.482272    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5497575d9c-lm8wr_b91cefe7-2e73-4f49-8d39-1187617dc044/router/0.log"
Oct 31 04:18:17.700463 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:17.700355    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-649697ffd5czs5l_6832d278-dba5-4b0d-af12-726ea83e1764/istio-proxy/0.log"
Oct 31 04:18:20.308861 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.308830    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ft2z2_de7d30b6-5160-4f1b-a67b-a0c92f02e0fa/serve-healthcheck-canary/0.log"
Oct 31 04:18:20.839539 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.839503    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-6vdn7_9a8b9637-0546-43b8-b5b4-c67bce1b0345/istio-proxy/0.log"
Oct 31 04:18:20.858727 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.858672    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-b8m6d_aee37c72-f7bb-459c-9f06-df7401cc1daf/istio-proxy/0.log"
Oct 31 04:18:20.894325 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.894290    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-j2gqr_bc41b4e8-af8b-420c-960d-8f948efbf80a/insights-operator/0.log"
Oct 31 04:18:20.896109 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.896085    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-j2gqr_bc41b4e8-af8b-420c-960d-8f948efbf80a/insights-operator/1.log"
Oct 31 04:18:20.919543 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.919510    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4crls_4e4684e1-6f21-46ca-8dec-b41f06386b69/kube-rbac-proxy/0.log"
Oct 31 04:18:20.941037 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.941010    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4crls_4e4684e1-6f21-46ca-8dec-b41f06386b69/exporter/0.log"
Oct 31 04:18:20.966373 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.966328    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4crls_4e4684e1-6f21-46ca-8dec-b41f06386b69/extractor/0.log"
Oct 31 04:18:20.979144 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.979114    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-769df7846c-tnsvh_a50c6120-b67e-4ca2-a369-5ae4b01d5fdb/istio-proxy/0.log"
Oct 31 04:18:20.993036 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:20.993008    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-64dd656c99-j9sjg_3afa466d-1d28-4af9-a622-b3be1061dd07/istio-proxy/0.log"
Oct 31 04:18:21.046267 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:21.046236    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5497575d9c-lm8wr_b91cefe7-2e73-4f49-8d39-1187617dc044/router/0.log"
Oct 31 04:18:21.150061 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:21.149973    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_periodic-gathering-xks58-zljvf_1fbddb03-ff9c-4eb1-b8de-b983c9889372/insights-gathering/0.log"
Oct 31 04:18:23.469122 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:23.469088    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-4ncnz_dc9fd9a3-bd12-4f19-bfa1-3cabdad01085/manager/0.log"
Oct 31 04:18:23.491837 ip-10-0-129-132 kubenswrapper[5098]: I1031 04:18:23.491806    5098 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-55975df5cc-4ncnz_dc9fd9a3-bd12-4f19-bfa1-3cabdad01085/copy-manifests/0.log"